IATI Data Store (11-11-18 onwards) ToR specifications & clarifications
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

View only
 
 
ABCDEFGHIJKLMN
1
TaskPhase 1ZZ Requirement questionTT feedback to ZZData validation service dependentDeployed by ZZ and testableAcceptance Test by TT
2
Phase 1: Extract, Transform and Load
3
A1.1DS should poll the IATI Registry continuously for updated or new files. Metadata should be stored and made available on the results of each poll (the list of files to be fetched) and each attempted fetch.Needs exact specification, too broad. Polling is clear, rest needs clarification. Update: import metadata, status of pull of org XML data and be able to feedback to publishers / user.Yes, this is delivered.OK
4
A1.2DS should access Registry validation logs for each file. (There is no need for DS to perform validation.) Failed files should be ignored. Activities in files with non- critical validation errors should be processed. Only files are fetched which have a current IATI registry validation log.Are Registry validation log available? Where? What format?Needs redefining: Validation will come from the validation service (yet to be built) datastore decides what to do based on the validation logsYesIn progressPENDING sync between Z&Z and D4D
5
A1.3Data should be transformed to the latest version of the standard (e.g. Version 2 standardisation of codes, narrative elements, location elements, vocabulary attributes, etc.) [In future, after the next major upgrade, there will be a need for data to be accessed in the two latest versions of the standard.](1) OIPA will not be supporting IATI 1.X data on the latest Python 3 branch
(2) Data transformation to the latests needs to be defined, too broad for single implementation specification. (a.i. one off or a permanent function? The latter seems out of scope)
(1) Validation service will no longer accept V1
(other) Some questions to go to the board about what data makes it to the DS. Need to explore writing a script to upgrade historic data from v1 to v2, discuss with the board.
Yes, this is delivered OK
6
A1.4Monetary values should be stored in the original currency and US Dollars (calculated using historical IMF rates for the relevant value dates).Where are the historical IMF rates located, in what format are they available and how will they made available? OIPA uses IMF: https://github.com/zimmerman-zimmerman/OIPA/pull/251/commits/2a8262bd6e3ca43558f509b32200f440a1839066SV: IMF has XML access. TT will check with Board.Yes, this is delivered.
7
A1.5A copy of the most recent original xml should be stored for each activity.Yes, this is delivered.OK
8
A1.6Processing and error handling during ETL needs to handle known complexities, including the reporting of the same activity twice, temporarily missing activities, etcDefine etc. Needs a clear final specification before implementation. Separate document required for defining thisYes, this is delivered. OK
9
A1.7All logic employed in transformation processes must be clearly and transparently documented for both developers and end users.Yes, this is delivered.TO VERIFY DOCS
10
A1.8Metadata should be maintained that records all validation errors (via Registry) and transformation processes for each activity.See question on A1.2No longer requiredyes
11
A1.9
Data should be stored in a manner that maximises speed and flexibility of query and access.
This is very / too broad. Needs exact requirementThis should be something that is done in a discuss post so we open this up to community conversation, only after we have drawn up our own metricsYes, this is delivered.OK
12
13
Phase 1: Filters Querying should be possible using the following filters:
14
B1.1Last updated date and timeYes, this is delivered.
15
B1.2All elements of the standardYes, this is delivered.
16
B1.2.1All elements of the standard Version 2.0.3Yes, this is delivered.
17
B1.2.2All elements of the standard Version 2.0.2Yes, this is delivered.
18
B1.2.3All elements of the standard Version 2.0.1Yes, this is delivered.
19
B1.3Free text search across title, description or entire activitySee A1.9See A1.9Yes, this is delivered.OK
20
B1.4Pre-processed aggregations and calculations fields, including: US Dollar monetary values; CRS Sectors; Years (from dates)Need a list of final aggregations requirements. Lets check OIPA endpoints first and see if they suffice. Check documentation.Yes, this is delivered.
21
22
Phase 1: Developer Output
23
C1.1Developer queries should result in the appropriately filtered delivery of entire standardised activities in XML formatYes, this is delivered.OK
24
C1.2Developer queries should result in the appropriately filtered delivery of entire standardised activities in JSON format.Yes, this is delivered.OK
25
C1.3Developer queries should result in the appropriately filtered delivery of entire standardised activities in original (untransformed) XML. Yes, this is delivered.
26
C1.4The standardised output should follow the standard IATI XML as close as possible.Yes, this is delivered.OK
27
28
Phase 1: API
29
D1.1Any products making use of the DS, including the DS query interface itself, may only communicate with the datastore via the API.Yes, this is delivered.OK
30
D1.2The API must be RESTful, must be versioned, must use HTTPSYes, this is delivered.OK
31
D1.3Results should be deliverable in full.Not sure if this an industry standard, will heavily impact application and will impact offering from H3.1
32
D1.4Results should be deliverable paginatedYes, this is delivered.OK
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...