TicketNetwork Feed Processor Specs
Feed Import Steps
General Updates Needed
Additional Notes
==================================================================
FILE AND FOLDER STRUCTURE
/home/abenity
/data
/credentials
/ticket_network_ftp.json
/log
/feed_error_tn.log
/feed_parsing_tn.log
/libraries
/ticket_network
/data
/category_mappings_tn.csv
/performer_id_blacklist_tn.csv
/geo_code_mapping_tn.csv
/examples
/parse_feed.php
/test_feed_tn.csv
/src
/Tn.php
/tests
/autoload.php
/TnTest.php
/composer.json
/readme.md
/build.xml
/phpdoc.dist.xml
/phpunit.xml.dist
==================================================================
==================================================================
CONSTANTS
SITE_ROOT = /home/abenity
APP_ROOT = /home/abenity/applications/discount-program
FEED_FOLDER = ticket_network
FEED_ABV = Tn
PROVIDER_ID = 748
ERROR_LOG = /home/abenity/data/log/feed_error_tn.log
==================================================================
FTP ACCOUNT INFO
FTP: feeds.ticketnetwork.com
UID: pf-2727
PWD: 2727zx$$
DIR: TNDataFeed
==================================================================
EVENTS CSV FEED FILENAME
TicketNetworkDataFeed-Events.csv.zip
TicketNetworkDataFeed-Events.csv
==================================================================
CATEGORY SHEET URL
https://docs.google.com/spreadsheets/d/1Ny8NetuofSqIPjDDm7LNcpr3I86HO-pDbPFcSEKGGTI/export?format=csv
==================================================================
==================================================================
PERFORMANCE REPORT DATA ITEMS
- Script filename*
- XML filename*
- Total run time of script*
- update_vendor_count
- new_vendor_count
- update_offer_count
- new_offer_count
- update_location_count
- new_location_count
*Included in the parse_feed.php file with the “update report with script run time xml filename” item
==================================================================
NEW FLEXOFFERS CATEGORY ASANA MESSAGE
TITLE:
Add <PCat> to the TN Events Category List
DESCRIPTION:
The TicketNetwork Events Feed contained a new category.\n\n
The feed category list needs to be updated to include the <PCat>. The category ID to add is: <PCatID>\n\n"
Please edit https://docs.google.com/spreadsheets/d/1Ny8NetuofSqIPjDDm7LNcpr3I86HO-pDbPFcSEKGGTI/edit#gid=0
==================================================================
DOWNLOAD CATEGORY SPREADSHEET AS CSV
The spreadsheet for creating the category CSV (category_mappings_tn.csv) is being maintained as a Google Sheet. We need to download the spreadsheet as a CSV using the export via URL function (see: http://stackoverflow.com/questions/21189665/new-google-spreadsheets-publish…).
- Once downloaded the new CSV will need to be named according to the feed processor file naming convention (category_mappings_tn.csv)
- The CSV will need to be saved in the current feed processor 'data' directory (data/category_mappings_tn.csv)
- The Abenity server has "allow_url_include" disabled so CURL will need to be used
- The CSV or anything imported from the CSV needs to be properly sanitized and treated as if it could be malicious. And the file that is downloaded and saved needs to be verified as a CSV file
- Name the function "downloadGoogleSheetCSV"
- We will need to pass the following parameters to the function: URL of the file to be pulled - name of the file to be created, destination directory for created file
- The function needs to be called as an initial step in parse_cj_feed.php, after the objects are created and before the data feed file is downloaded.
- If the process (download, file verification, name, and save) fails then a notification email needs to be sent via the sendAlert function in the FeedProcessor class and and the previously downloaded category CSV should be used to for the current run of the feed processor.
- The failed download exception handling needs to incorporate 404
- The failed download exception handling needs to incorporate a bad spreadsheet link / change in google permissions because those google notification pages will be downloaded as a file by CURL and saved as a CSV.
- This function (downloadGoogleSheetCSV) already exists in the FeedProcessor Class
==================================================================
DOWNLOAD TN STATES CSV
TicketNetwork maintains a CSV that maps states and country ID to their respective names and abreviations (TicketNetworkDataFeed-States.csv) that is in the same directory on the FTP server as the events CSV. The states CSV is needed for providing the proper two character ISO country and state codes in the offer listing data.
- Once downloaded the new CSV will need to be named according to the feed processor file naming convention (geo_code_mapping_tn.csv)
- The CSV will need to be saved in the current feed processor 'data' directory (data/geo_code_mapping_tn.csv)
- The download function needs to be called as an initial step in parse_feed.php, after the objects are created and before the data feed file is downloaded.
- If the process (download, rename, and save) fails then a notification email needs to be sent via the sendAlert function in the FeedProcessor class
- If the process (download, rename, and save) fails then log the error to the unified error log file (abenity/log/feed_error_tn.log)
- If the process (download, rename, and save) fails then the previously downloaded countries CSV should be used to for the current run of the feed processor.
==================================================================
STATES CSV VERIFICATION
This is NOT currently included in the TicketNetwork feed processor but I think that it would be a good thing to include to verify that the file is a CSV and that format of the CSV file matches the scheme used in the feed processor. If the validation fails then an alert email should be sent and the error logged in the TN error log (abenity/log/feed_error_tn.log)
There are 4 potential items to validate...
- File Extention (.csv)
- MIME Type of the File (text/csv, text/plain, application/csv, text/comma-separated-values, application/excel, application/vnd.ms-excel, application/vnd.msexcel, text/anytext, application/octet-stream, application/txt)
- The number of collomns in the file (6)
- The collumn header names and/or order in the file (StateID, StateShortDesc, StateLongDesc, CountryID, CountryShortDesc, CountryLongDesc)
Links
- https://github.com/alimanfoo/csvvalidator
- https://github.com/theodi/csvlint
- https://pythonhosted.org/chkcsv/
- http://stackoverflow.com/questions/6654351/check-file-uploaded-is-in-csv-format
- https://github.com/zumba/csv-policy/blob/master/src/Zumba/CsvPolicy/Validator.php
- https://github.com/javilumbrales/csv_file_validation
- https://github.com/goodby/csv
==================================================================
UNCOMPRESS CSV FILE
The feed CSV will need to be uncompressed from the downloaded .ZIP file.
- Use the uncompressFeed function in the FeedProcessor Class
- The .ZIP file needs to be removed after the CSV has been successfully extracted
==================================================================
EVENTS CSV VERIFICATION
This is NOT currently included in the TicketNetwork feed processor but I think that it would be a good thing to include to verify that the file is a CSV and that format of the CSV file matches the scheme used in the feed processor. If the validation fails then an alert email should be sent and the error logged in the TN error log (abenity/log/feed_error_tn.log)
There are 4 potential items to validate...
- File Extention (.csv)
- MIME Type of the File (text/csv, text/plain, application/csv, text/comma-separated-values, application/excel, application/vnd.ms-excel, application/vnd.msexcel, text/anytext, application/octet-stream, application/txt)
- The number of collomns in the file (28)
- The collumn header names and/or order in the file (EventID, Event, PerformerID, Performer, Venue, VenueID, VenueStreetAddress, DateTime, PCatID, PCat, CCatID, CCat, GCatID, GCat, City, State, StateID, Country, CountryID, Zip, TicketsYN, MinPrice, MaxPrice, IMAGEURL, URLLink, NumOrders, NumTicketsSold, EventUpdateDate)
Links
- https://github.com/alimanfoo/csvvalidator
- https://github.com/theodi/csvlint
- https://pythonhosted.org/chkcsv/
- http://stackoverflow.com/questions/6654351/check-file-uploaded-is-in-csv-format
- https://github.com/zumba/csv-policy/blob/master/src/Zumba/CsvPolicy/Validator.php
- https://github.com/javilumbrales/csv_file_validation
- https://github.com/goodby/csv
==================================================================
PERFORMER BLACKLIST
There are certain Performers that, for a variety of reasons, we do NOT want their events processed and saved. This is essentially the same as the Provider ID Blacklist that we did for CJ and FO but using Performer ID <PerformerID> instead.
- We need to filter all offers through the TN Performer ID blacklist.
- The blacklist needs to be a CSV pulled in as an array.
- The blacklist CSV needs to be in the repo data folder (data/performer_id_blacklist_tn.csv)
- This function (checkBlacklist) already exists in the FeedProcessor Class
==================================================================
EVENT KEYWORD BLACKLIST
The keyword blacklist filter support multiple datapoints passed to the function as an array. We need to pass the event name <Event> and venue name <Venue> for TN.
NOTES
- The filter will work similar to the offer title whitelist in that the the keyword list will be maintained in CSV file that we'll pull into an array.
- The keyword blacklist CSV is in the feedprocessor repo data folder (data/feedprocessor_keyword_blacklist.csv)
- The checkBlacklist accept an argument to define csv file containing the blacklist.
- The data points to check should be passed to the method as an array that we'll loop through to check against the blacklist
- This function (checkBlacklist) already exists in the FeedProcessor Class
==================================================================
EVENT TICKETS FLAG
We need to verify that the event has tickets available, if the event does NOT have tickets then it will be skipped and we'll move on to the next event.
- Currently this check is being performed AFTER all data normalization is completed and AFTER the listing array has been built (see TicketNetwork.class.php).
- However, this check should be completed BEFORE those two steps along with the other criteria (whitelist, blacklist, etc.)
==================================================================
ABENITY VENDOR ID SEARCH
- Search for Abenity provider_vendor_id by [AdvertiserID] and provider_id
- This function (checkProviderVendorID) already exists in the FeedProcessor Class
==================================================================
MAP EVENT COUNTRY CODE
The Abenity listing aray require the country be defined using ISO two character country codes (i.e. US or CA). The TN feed contains the long form country name and a CountryID that is mapped using the TicketNetworkDataFeed-States.csv they provide.
- Use the CountryID to map the two character country code
- Read in the TN states CSV (TicketNetworkDataFeed-States.csv) that was downloaded and saved as (data/country_code_mapping_tn.csv) as an array
- If the CSV does not contain a match for CountryID then the event should be skipped
- This function exists in the current TicketNetwork class but will need some minor modifications including the CSV column numbers (the numbers used in the current function were from a CSV with a different schema)
==================================================================
MAP EVENT STATE CODE
The Abenity listing aray require the states to be defined using ISO two character state codes (i.e. AL or OH). The TN feed contains the long form state name and a StateID that is mapped using the TicketNetworkDataFeed-States.csv they provide.
- Use the StateID to map the two character State code/abbreviation
- Read in the TN states CSV (TicketNetworkDataFeed-States.csv) that was downloaded and saved as (data/country_code_mapping_tn.csv) as an array
- If the CSV does not contain a match for StateID then the event should be skipped
- A version of this function exists in the current TicketNetwork class for mapping country codes but will need some minor modifications including the CSV column numbers (the numbers used in the current function were for country codes using a CSV with a different schema)
==================================================================
FORMAT US ZIP CODES
The abenity_geodata database that is used for cross referencing long/lad based off zip code does note contain the leading zeroes on US zipcodes.
- If the event country code is US then the leading zeroes need to be removed from the zip code
- Events with any other country code should use the provided zip code as is
==================================================================
ASSIGN ABENITY CATEGORY ID
- The same category mapping process as CJ and FO
- Map based on TN <PCatID>
- Use the downloaded Category CSV (data/category_mappings_tn.csv)
- This function (mapProviderCategoryID) exists in the Feed Process Class
==================================================================
FORMAT TICKET PRICE
The ticket price <MinPrice> will be used in both the offer title and offer description but needs to be properly formated first.
- Standard USD format
- Two decimal places
- Thousands Comma separated
==================================================================
OFFER TITLE FORMAT
All offer titles need to be standardized copy dynamically populated with event data from the CSV. Use the following format:
"10% off premium and sold-out seats to see <Event> starting at just $<FORMATTED_MinPrice>!"
==================================================================
OFFER DESCRIPTION FORMAT
All offer descriptions need to be standarized copy dynamically populated with event data from the CSV. Use the following format:
"Save 10% on premium and sold-out seats to see <Event> starting at just $<FORMATTED_MinPrice>. Purchases made through the Ticket Center are sourced through TicketNetwork.com, the leading online exchange for after-market event tickets."
==================================================================
OFFER REDEMPTION URL FORMAT
The offer redemption URL will use a formatted version of the event's <URLLink>. The following two modification need to be made...
- Replace the <INSERT-YOUR-SITE-NAME-HERE> placeholder in the URLLink with "tickets.abenity.com"
- The URLLink needs to be appended with the abenity user ID ?sid=<abty:user_id>
- e.g. http://tickets.abenity.com/ResultsTicket.aspx?evtid=2639881?sid=<abty:user_id>
==================================================================
NORMALIZE START DATE
The start date format needs to formatted and a default set if a date isn't provided
- The start date needs to be converted to "Y-m-d" format.
- If the feed does NOT contain a start date then then the start date needs to be set to the current date.
- This function (normalizeStartDate) already exists in the FeedProcessor Class
==================================================================
NORMALIZE EXPIRATION DATE
The expiri/end date format needs to formatted and a default set if a date isn't provided
- The expiri/end date needs to be converted to "Y-m-d" format.
- If the feed does NOT contain a expiri/end date then then the expiri/end date needs to be set to one year from the current date
- This function (normalizeEndDate) already exists in the FeedProcessor Class
==================================================================
ADDITIONAL OFFER SETTINGS
The offer deal field and the items listed below need to be set as true or false with a boolean. The correct settings are specified in the Abenity Field Mapping Spreadsheet as well.
$listing['offer'][0]['universal'] = 0;
$listing['offer'][0]['status'] = 1;
$listing['offer'][0]['deal'] = 0;
$listing['offer'][0]['printable_coupon'] = '';
$listing['offer'][0]['mobile_coupon'] = '';
$listing['offer'][0]['card_presentation'] = '';
==================================================================
MISC SPECS
==================================================================