ETL with Python for Esploro Migration�
CASEY KOONS
SCHOLARLY COMMUNICATIONS LIBRARIAN
SUNY COLLEGE OF ENVIRONMENTAL SCIENCE AND FORESTRY
Who am I
Many Hats
Who am I to tell you about Python?
Project Based Learning�is a fine way to learn.
What is this presentation about?
Python scripts I used to address issues with data migration into Esploro.
Walking through files and directories
Parsing and editing xml files
Transforming data (dates, names)
Parsing and editing csv files
A review of Lists and Loops
Two Stories
The Bepress Digital Commons to Esploro Migration
Esploro Bulk Upload CSV file construction
Different workflow
T
E
L
Esploro Migration
Digital Commons metadata challenge
Issues with Digital Commons Metadata
Information was missing
The metadata did not fit the Migration Form
Need to make�bulk changes�to many xml files.
OS Walk – Spooky Stuff
XML ElementTree
Combine the os.walk and etree to bulk edit!
Esploro Bulk CVS Upload
Lost ETDs
Source Data
Upload Format
Problems with Data
Professors' names entered in a single field – need first, last, etc.
Upload csv wants two different date types
NameParser
pip install nameparser
Python datetime
Better date math with dateutil
pip install python-dateutil
Constructing a CSV file with Python
Review of what Python can do:
WALK THROUGH FILE STRUCTURES WITH OS.WALK
READ AND WRITE TO XML FILES WITH XML.ETREE.ELEMENTTREE
PARSE NAME STRINGS TO FIRST, LAST, ETC, WITH NAMEPARSER.HUMANNAME
PARSE, FORMAT AND CALCULATE DELTAS IN DATES WITH DATETIME & DATEUTIL
READ, APPEND AND CONSTRUCT CSV FILES WITH CSV