TicketNetwork Feed Processor Specs

Feed Import Steps

Pre-Processing Steps (start timer, define roots, feed constants, connect to database, load json credential, define server credentials, include classes, create objects)
Download Category Spreadsheet as CSV (see CATEGORY SHEET URL and DOWNLOAD CATEGORY SPREADSHEET AS CSV notes)
Download the TicketNetwork States CSV (see DOWNLOAD TN STATES CSV)
Verify the State CSV File (see STATES CSV VERIFICATION notes below), if failed send alert email and log error
Download TicketNetwork Events .ZIP File Containing the Events Feed CSV File (see FTP ACCOUNT INFO and CSV FEED FILENAME if failed send alert email and log in the error log.
Uncompress the TicketNetwork Events .ZIP File Containing the Events Feed CSV File (see UNCOMPRESS CSV FILE notes), if failed send alert email and log in error log.
Verify the Events Feed CSV File (see EVENTS CSV VERIFICATION notes below), if failed send alert email and log error
Parse CSV Row feed into local PHP $feedRow array
Filter Promotions Through Performer Blacklist (see PERFORMER BLACKLIST notes)
Filter Event Name <Event> and Venue Name <Venue> Through Keyword Blacklist (see EVENT KEYWORD BLACKLIST notest below)
Verify That the Event has Tickets (see EVENT TICKETS FLAG notes below)

Lookup Abenity vendor_id by <EventID> and provider_id (see ABENITY VENDOR ID SEARCH notes), if it's an existing vendor use the returned vendor ID.
Map the Event’s Country Code (see MAP EVENT COUNTRY CODE notes below)
Map the Events State Code (see MAP EVENT STATE CODE notes below)
Strip the Leading Zeroes from US Zip Codes (see FORMAT US ZIP CODES notes below)
Assign Abenity Category ID from downloaded CSV (see item #2) mapped by <CategoryID> (see ASSIGN ABENITY CATEGORY ID notes)
If the <CategoryID> is NEW and NOT in the category CSV trigger an Asana task (using the createAsanaTask in the FeedProcessor class) to update category CSV with new <PCatID> (with duplicate Asana task error checking) and SKIP the current Promotion
Format the Ticket Price (see FORMAT TICKET PRICE notes below)
Build the Offer Title (see OFFER TITLE FORMAT notes below)
Build the Offer Description (see OFFER DESCRIPTION FORMAT notes below)
Build the Offer Redemption URL (see OFFER REDEMPTION URL FORMAT notes below)

Normalize Start Date <EventUpdateDate> (see NORMALIZE START DATE notes)
Normalize Expiration Date <DateTime> (see NORMALIZE EXPIRATION DATE notes)
Map XML Fields to Offer fields (see http://doc.beanstalkcreative.com/flexoffers-field-mapping)
The Offer Deal Flag Should be Set to FALSE (or ‘0’)and set standard offer setting (see ADDITIONAL OFFER SETTINGS notes)
Call save_listing to Add or Update Offer
Update report counts (updated vendors, new vendors, updated offers, new offers, updated locations, new locations)
Add 5 seconds to script execution time (set_time_limit)
Repeat for all CSV Rows
Delete CSV File
Log File Processing Specs & Send CRON Report Email
Send Notification Email IF parsing did NOT succeed

General Updates Needed

Update filenames and paths to be TicketNetork Specific
Update code commenting to be TicketNetork Specific
Update Email and Asana Task Verbiage to be TicketNetork Specific (see NEW FLEXOFFERS CATEGORY ASANA MESSAGE)
Make sure commenting, formatting, etc. is thorough that code is PSR 2.0 compliant.

Additional Notes

Use the Commission Junction Feed and FlexOffers Processor as the framework reuse as much code as possible.
Follow Standard Server File Structure, File Naming Conventions and Composer Package Library established for CJ
The FlexOffers Repo has been setup - https://goggans@bitbucket.org/abenity/ticket-network-library.git
The TicketNetwork repo contains two pertinent branches ‘master’ and ‘nov_refactor’.
The master branch contains the orginal work, much of which will need to be scrapped but will be good for reference and pulling some code that can be reused.
The ‘nov_refactor’ branch has all the old and non-usable files deleted as well as updates CSVs in the data directory and a recent feed CSV in the examples directory. The remaining files will need to be added/created.
All error logging should be consolidated to a single feed error log file (/data/log/feed_error_tn.log)
FTP / Server Credentials should be stored in a JSON file (/data/credentials/ticket_network_ftp.json)
Connect to the database using PDO
Use the testing mode (set to TRUE) during development testing
Use the FEED_ABV, and PROVIDER_ID constants anywhere in the parse_feed.php or Tn class (Tn.php) file instead of hard coding those items
Complete phpdoc blocks are required
Include clear and complete commenting throughout the code.
The library needs to work as a composer package
The following build processes need to be included: code sniffing, documentation generation, and unit testing with code coverage reports.
You don’t need to set up tests for 100% code coverage, but it needs at least the setup in place to build upon later. Don’t commit the report to the repo - it should be already git-ignored
The “core” code should be in the src folder and tests in the tests folder
You should be able to reuse composer.json, build.xml, phpdoc.dist.xml and phpunit.xml.dist from the CJ library with minimal changes to the package name, etc.

==================================================================

FILE AND FOLDER STRUCTURE

/home/abenity

/data

/credentials

/ticket_network_ftp.json

/log

/feed_error_tn.log

/feed_parsing_tn.log

/libraries

/ticket_network

/data

/category_mappings_tn.csv

/performer_id_blacklist_tn.csv

/geo_code_mapping_tn.csv

/examples

/parse_feed.php

/test_feed_tn.csv

/src

/Tn.php

/tests

/autoload.php

/TnTest.php

/composer.json

/readme.md

/build.xml

/phpdoc.dist.xml

/phpunit.xml.dist

==================================================================

CONSTANTS

SITE_ROOT = /home/abenity

APP_ROOT = /home/abenity/applications/discount-program

FEED_FOLDER = ticket_network

FEED_ABV = Tn

PROVIDER_ID = 748

ERROR_LOG = /home/abenity/data/log/feed_error_tn.log

==================================================================

FTP ACCOUNT INFO

FTP: feeds.ticketnetwork.com

UID: pf-2727

PWD: 2727zx$$

DIR: TNDataFeed

==================================================================

EVENTS CSV FEED FILENAME

TicketNetworkDataFeed-Events.csv.zip

TicketNetworkDataFeed-Events.csv

==================================================================

CATEGORY SHEET URL

https://docs.google.com/spreadsheets/d/1Ny8NetuofSqIPjDDm7LNcpr3I86HO-pDbPFcSEKGGTI/export?format=csv

==================================================================

PERFORMANCE REPORT DATA ITEMS

- Script filename*

- XML filename*

- Total run time of script*

- update_vendor_count

- new_vendor_count

- update_offer_count

- new_offer_count

- update_location_count

- new_location_count

*Included in the parse_feed.php file with the “update report with script run time xml filename” item

==================================================================

NEW FLEXOFFERS CATEGORY ASANA MESSAGE

TITLE:

Add <PCat> to the TN Events Category List

DESCRIPTION:

The TicketNetwork Events Feed contained a new category.\n\n

The feed category list needs to be updated to include the <PCat>. The category ID to add is: <PCatID>\n\n"

Please edit https://docs.google.com/spreadsheets/d/1Ny8NetuofSqIPjDDm7LNcpr3I86HO-pDbPFcSEKGGTI/edit#gid=0

==================================================================

DOWNLOAD CATEGORY SPREADSHEET AS CSV

The spreadsheet for creating the category CSV (category_mappings_tn.csv) is being maintained as a Google Sheet. We need to download the spreadsheet as a CSV using the export via URL function (see: http://stackoverflow.com/questions/21189665/new-google-spreadsheets-publish…).

- Once downloaded the new CSV will need to be named according to the feed processor file naming convention (category_mappings_tn.csv)

- The CSV will need to be saved in the current feed processor 'data' directory (data/category_mappings_tn.csv)

- The Abenity server has "allow_url_include" disabled so CURL will need to be used

- The CSV or anything imported from the CSV needs to be properly sanitized and treated as if it could be malicious. And the file that is downloaded and saved needs to be verified as a CSV file

- Name the function "downloadGoogleSheetCSV"

- We will need to pass the following parameters to the function: URL of the file to be pulled - name of the file to be created, destination directory for created file

- The function needs to be called as an initial step in parse_cj_feed.php, after the objects are created and before the data feed file is downloaded.

- If the process (download, file verification, name, and save) fails then a notification email needs to be sent via the sendAlert function in the FeedProcessor class and and the previously downloaded category CSV should be used to for the current run of the feed processor.

- The failed download exception handling needs to incorporate 404

- The failed download exception handling needs to incorporate a bad spreadsheet link / change in google permissions because those google notification pages will be downloaded as a file by CURL and saved as a CSV.

- This function (downloadGoogleSheetCSV) already exists in the FeedProcessor Class

==================================================================

DOWNLOAD TN STATES CSV

TicketNetwork maintains a CSV that maps states and country ID to their respective names and abreviations (TicketNetworkDataFeed-States.csv) that is in the same directory on the FTP server as the events CSV. The states CSV is needed for providing the proper two character ISO country and state codes in the offer listing data.

- Once downloaded the new CSV will need to be named according to the feed processor file naming convention (geo_code_mapping_tn.csv)

- The CSV will need to be saved in the current feed processor 'data' directory (data/geo_code_mapping_tn.csv)

- The download function needs to be called as an initial step in parse_feed.php, after the objects are created and before the data feed file is downloaded.

- If the process (download, rename, and save) fails then a notification email needs to be sent via the sendAlert function in the FeedProcessor class

- If the process (download, rename, and save) fails then log the error to the unified error log file (abenity/log/feed_error_tn.log)

- If the process (download, rename, and save) fails then the previously downloaded countries CSV should be used to for the current run of the feed processor.

==================================================================

STATES CSV VERIFICATION

This is NOT currently included in the TicketNetwork feed processor but I think that it would be a good thing to include to verify that the file is a CSV and that format of the CSV file matches the scheme used in the feed processor. If the validation fails then an alert email should be sent and the error logged in the TN error log (abenity/log/feed_error_tn.log)

There are 4 potential items to validate...

- File Extention (.csv)

- MIME Type of the File (text/csv, text/plain, application/csv, text/comma-separated-values, application/excel, application/vnd.ms-excel, application/vnd.msexcel, text/anytext, application/octet-stream, application/txt)

- The number of collomns in the file (6)

- The collumn header names and/or order in the file (StateID, StateShortDesc, StateLongDesc, CountryID, CountryShortDesc, CountryLongDesc)

Links

- https://github.com/alimanfoo/csvvalidator

- https://github.com/theodi/csvlint

- https://pythonhosted.org/chkcsv/

- http://stackoverflow.com/questions/6654351/check-file-uploaded-is-in-csv-format

- https://github.com/zumba/csv-policy/blob/master/src/Zumba/CsvPolicy/Validator.php

- https://github.com/javilumbrales/csv_file_validation

- https://github.com/goodby/csv

==================================================================

UNCOMPRESS CSV FILE

The feed CSV will need to be uncompressed from the downloaded .ZIP file.

- Use the uncompressFeed function in the FeedProcessor Class

- The .ZIP file needs to be removed after the CSV has been successfully extracted

==================================================================

EVENTS CSV VERIFICATION

There are 4 potential items to validate...

- File Extention (.csv)

- The number of collomns in the file (28)

- The collumn header names and/or order in the file (EventID, Event, PerformerID, Performer, Venue, VenueID, VenueStreetAddress, DateTime, PCatID, PCat, CCatID, CCat, GCatID, GCat, City, State, StateID, Country, CountryID, Zip, TicketsYN, MinPrice, MaxPrice, IMAGEURL, URLLink, NumOrders, NumTicketsSold, EventUpdateDate)

Links

- https://github.com/alimanfoo/csvvalidator

- https://github.com/theodi/csvlint

- https://pythonhosted.org/chkcsv/

- http://stackoverflow.com/questions/6654351/check-file-uploaded-is-in-csv-format

- https://github.com/zumba/csv-policy/blob/master/src/Zumba/CsvPolicy/Validator.php

- https://github.com/javilumbrales/csv_file_validation

- https://github.com/goodby/csv

==================================================================

PERFORMER BLACKLIST

There are certain Performers that, for a variety of reasons, we do NOT want their events processed and saved. This is essentially the same as the Provider ID Blacklist that we did for CJ and FO but using Performer ID <PerformerID> instead.

- We need to filter all offers through the TN Performer ID blacklist.

- The blacklist needs to be a CSV pulled in as an array.

- The blacklist CSV needs to be in the repo data folder (data/performer_id_blacklist_tn.csv)

- This function (checkBlacklist) already exists in the FeedProcessor Class

==================================================================

EVENT KEYWORD BLACKLIST

The keyword blacklist filter support multiple datapoints passed to the function as an array. We need to pass the event name <Event> and venue name <Venue> for TN.

NOTES

- The filter will work similar to the offer title whitelist in that the the keyword list will be maintained in CSV file that we'll pull into an array.

- The keyword blacklist CSV is in the feedprocessor repo data folder (data/feedprocessor_keyword_blacklist.csv)

- The checkBlacklist accept an argument to define csv file containing the blacklist.

- The data points to check should be passed to the method as an array that we'll loop through to check against the blacklist

- This function (checkBlacklist) already exists in the FeedProcessor Class

==================================================================

EVENT TICKETS FLAG

We need to verify that the event has tickets available, if the event does NOT have tickets then it will be skipped and we'll move on to the next event.

- Currently this check is being performed AFTER all data normalization is completed and AFTER the listing array has been built (see TicketNetwork.class.php).

- However, this check should be completed BEFORE those two steps along with the other criteria (whitelist, blacklist, etc.)

==================================================================

ABENITY VENDOR ID SEARCH

- Search for Abenity provider_vendor_id by [AdvertiserID] and provider_id

- This function (checkProviderVendorID) already exists in the FeedProcessor Class

==================================================================

MAP EVENT COUNTRY CODE

The Abenity listing aray require the country be defined using ISO two character country codes (i.e. US or CA). The TN feed contains the long form country name and a CountryID that is mapped using the TicketNetworkDataFeed-States.csv they provide.

- Use the CountryID to map the two character country code

- Read in the TN states CSV (TicketNetworkDataFeed-States.csv) that was downloaded and saved as (data/country_code_mapping_tn.csv) as an array

- If the CSV does not contain a match for CountryID then the event should be skipped

- This function exists in the current TicketNetwork class but will need some minor modifications including the CSV column numbers (the numbers used in the current function were from a CSV with a different schema)

==================================================================

MAP EVENT STATE CODE

The Abenity listing aray require the states to be defined using ISO two character state codes (i.e. AL or OH). The TN feed contains the long form state name and a StateID that is mapped using the TicketNetworkDataFeed-States.csv they provide.

- Use the StateID to map the two character State code/abbreviation

- Read in the TN states CSV (TicketNetworkDataFeed-States.csv) that was downloaded and saved as (data/country_code_mapping_tn.csv) as an array

- If the CSV does not contain a match for StateID then the event should be skipped

- A version of this function exists in the current TicketNetwork class for mapping country codes but will need some minor modifications including the CSV column numbers (the numbers used in the current function were for country codes using a CSV with a different schema)

==================================================================

FORMAT US ZIP CODES

The abenity_geodata database that is used for cross referencing long/lad based off zip code does note contain the leading zeroes on US zipcodes.

- If the event country code is US then the leading zeroes need to be removed from the zip code

- Events with any other country code should use the provided zip code as is

==================================================================

ASSIGN ABENITY CATEGORY ID

- The same category mapping process as CJ and FO

- Map based on TN <PCatID>

- Use the downloaded Category CSV (data/category_mappings_tn.csv)

- This function (mapProviderCategoryID) exists in the Feed Process Class

==================================================================

FORMAT TICKET PRICE

The ticket price <MinPrice> will be used in both the offer title and offer description but needs to be properly formated first.

- Standard USD format

- Two decimal places

- Thousands Comma separated

==================================================================

OFFER TITLE FORMAT

All offer titles need to be standardized copy dynamically populated with event data from the CSV. Use the following format:

"10% off premium and sold-out seats to see <Event> starting at just $<FORMATTED_MinPrice>!"

==================================================================

OFFER DESCRIPTION FORMAT

All offer descriptions need to be standarized copy dynamically populated with event data from the CSV. Use the following format:

"Save 10% on premium and sold-out seats to see <Event> starting at just $<FORMATTED_MinPrice>. Purchases made through the Ticket Center are sourced through TicketNetwork.com, the leading online exchange for after-market event tickets."

==================================================================

OFFER REDEMPTION URL FORMAT

The offer redemption URL will use a formatted version of the event's <URLLink>. The following two modification need to be made...

- Replace the <INSERT-YOUR-SITE-NAME-HERE> placeholder in the URLLink with "tickets.abenity.com"

- The URLLink needs to be appended with the abenity user ID ?sid=<abty:user_id>

- e.g. http://tickets.abenity.com/ResultsTicket.aspx?evtid=2639881?sid=<abty:user_id>

==================================================================

NORMALIZE START DATE

The start date format needs to formatted and a default set if a date isn't provided

- The start date needs to be converted to "Y-m-d" format.

- If the feed does NOT contain a start date then then the start date needs to be set to the current date.

- This function (normalizeStartDate) already exists in the FeedProcessor Class

==================================================================

NORMALIZE EXPIRATION DATE

The expiri/end date format needs to formatted and a default set if a date isn't provided

- The expiri/end date needs to be converted to "Y-m-d" format.

- If the feed does NOT contain a expiri/end date then then the expiri/end date needs to be set to one year from the current date

- This function (normalizeEndDate) already exists in the FeedProcessor Class

==================================================================

ADDITIONAL OFFER SETTINGS

The offer deal field and the items listed below need to be set as true or false with a boolean. The correct settings are specified in the Abenity Field Mapping Spreadsheet as well.

$listing['offer'][0]['universal'] = 0;

$listing['offer'][0]['status'] = 1;

$listing['offer'][0]['deal'] = 0;

$listing['offer'][0]['printable_coupon'] = '';

$listing['offer'][0]['mobile_coupon'] = '';

$listing['offer'][0]['card_presentation'] = '';

==================================================================

MISC SPECS

==================================================================