3 of 26

1) Why is Data Management Important?

How often do we get “clean” data?

Extremely rare (never myself)

Gap in methods/statistics education

Taught the “how” of analysis, not management of data to get to that point

Typical cause of delay

For newcomers or basic users, could take weeks or months to get data ready

Enables use of different kinds of statistical analyses

4 of 26

2-A) Sample Data

NLSY97

Longitudinal sample of American youth, ages 12-16 as of Wave 1

Motivating Question

What is the relationship between employment (dichotomous, weeks worked) and the variety of crimes (0-17 different types of self-reported crimes) a youth reports in any given wave?

Data

Waves 1-3

Variables

Employment:

Yes/No
Weeks worked (0 – max)

Crime Variety Score
Demographics
Job Satisfaction
And more…

5 of 26

2-B) Some Preliminary Definitions

Data in “Wide” Format

Data in “Long” Format

id	wave	variety
24	1
24	2
1341	1
1341	2
4755	1
4755	2

id	variety1	variety2
24
1341
4755

6 of 26

2-C) Some Preliminary Definitions

Information to Know

Observations are simply persons, matrix is as wide as the number of covariates in the data set.
Helpful to use a numerical suffix to denote observation sequences, hence the variety1 and variety2 depicted here.

As long as variables named this way, can use shortcuts in reshape commands (as well as other commands, too).
For example, if I want to compute descriptive statistics across variety scores, I can use an asterisk to denote that I want them for all variety variables

tabstat variety*, stats(mean N) column(statistics)

Data in “Wide” Format

id	variety1	variety2
24
1341
4755

7 of 26

2-D) Some Preliminary Definitions

Information to Know

Observations are now person-waves, matrix is skinnier now, but much taller as the sample size is now N * t, or the number of original cases times the number of waves.
No longer have use for suffixes, but that means descriptive statistics are slightly more difficult to obtain by wave.

bysort wave: sum variety, detail

Could also compute person-means across waves:

bysort id: sum variety, detail
bysort id: egen varmean=mean(variety)

Data in “Long” Format

id	wave	variety
24	1
24	2
1341	1
1341	2
4755	1
4755	2

8 of 26

3) Basic Functions to Know – A Short, Incomplete List

A) By, sort and Bysort

B) Egen

C) Reshape

D) Gsort

E) Lags and leads

F) Carryforward

9 of 26

3-A) Basic Functions to Know: By & Bysort

Function

Various – runs another Stata function within groups defined with “by” or “bysort”

In practice, “by, sort” and “bysort” are the same
If I want to see descriptive statistics of variety1 by race, I would write:

bysort race: sum variety1, detail
Or – by race, sort: sum variety1, detail

Helpful Hint

Suppose you need or want to sort by a variable but you do not want statistics created according to it – for generating a mean crime variety score for the first 3 waves, you would write this:

bysort nlsyid (wave): egen varmean3=mean(variety) if wave<=3
Essentially, I want to be sure the data is sorted by person-waves, but do not want a different mean for every person-wave, which is what ”bysort nlsyid wave: egen varmean3=mean(variety) if wave<=3” would do

10 of 26

3-B) Basic Functions to Know: Egen

Function

Will run a number of mathematical procedures over a set of pre-defined variables

When data are in long format, egen tends to be helpful when used with the “bysort” function
Say I want cluster means for gross household income over time, I would write:

bysort nlsyid: egen hhinc_mean=mean(hhinc)

Helpful Hint

Perhaps the most useful egen command when dealing with panel data is “tag” – here’s what it looks like in action:

I have data in person-observations and want to highlight just one row per person to put together static descriptives (sex, race, etc…) – egen id_tag=tag(nlsy_id)
Results in the first row per person having a value of 1 for id_tag and all other rows will have 0s.
Also very helpful when you want to tag unique observations within person (e.g., arrest cycles within person ids)

Keep in mind that egen will, by default, treat missing values as 0s – this can be rectified by adding a conditional statement to the end of the function (true for many egen functions, not all)

11 of 26

3-C) Basic Functions to Know: Reshape

Function

Will switch data from long to wide (and vice versa)

Suppose I have data in “wide” format that I need to have in “long” to run panel data analyses, I would use “reshape long ….” to accomplish this
reshape long varnames, i(person id variable) j(panel series varname)

Helpful Hint

It’s always good to keep a version of your data in each format, just in case you need the data in a different format for variable creation or some type of analysis
Also, be sure to reshape the data in a different do file than the one you use to run analyses – if you mess something up, you’ll have to re-run the reshape again if you don’t (I will break this rule later)

12 of 26

3-D) Basic Functions to Know: Gsort

Function

Alternative to “sort” allows you to sort variables in ascending or descending order

Say you need to sort one variables in ascending order and another in descending order – gsort allows you to do this

gsort +nlsyid –wave
This will sort nlysid in ascending order (+) and wave in descending order (-)

Helpful Hints

Often helpful when putting data in their initial order, or when computing variables within IDs that are sensitive to sort order

13 of 26

3-E) Basic Functions to Know: Lags and Leads

Function

Allow you to reference observations before (lag) or after (lead) a certain observation of interest

Let’s say I want to create a difference between this and last year’s crime variety score but my data are now in long format – no need to worry, a lag function can solve that easily
gen vardif=variety-variety[_n-1] if nlsyid==nslyid[_n-1] & year!=1

Tells Stata to generate a difference value for each variety score comparison (2 to 1, 3 to 2, 4 to 3, etc…) as long as nlsyid is the same for the comparison and it is not the first observation (no difference exists there, of course).

Helpful Hint

Lagged values can be helpful in a number of ways, but be mindful of the fact that they can create some pretty serious multicollinearity problems at times

14 of 26

3-F) Basic Functions to Know: Carryforward

Function

Allows you to copy and replace values down rows

Suppose I have a variable measured in only the first wave, but I want to copy it into the following rows – carryforward can do that easily, but you need to be careful

bysort person_id: carryforward varname, replace

Helpful Hint

Always use carryforward with the by, sort or bysort command, otherwise it will copy values into rows for other IDs

15 of 26

4) Data Exercises with Basic Functions

Or, the complicated nature of things that should,

by all means, be easier than they are.

16 of 26

5) Advanced Functions to Know – Another Short, Incomplete List

A) Import and Export

B) Merge

C) Global and Local

D) Foreach and Forvalues

E) While

F) Statsby

17 of 26

5-A) Advanced Functions to Know: Import and Export

Function

Allows you to port various types of data into or out of Stata (most helpful with Excel files)

Let’s say someone you collaborate with uses a different data analysis program than you do – this means you will have to share a file that is universally recognized by both programs – often, this is some type of Microsoft Excel file (.csv; .xls; .xlsx)
Luckily, the import function now accepts excel files with multiple sheets, so you do not always have to use a .csv (like I used to).

import excel using “file path & name here”, sheet(“sheetname here”) other options…..
export excel using “file path & name here”,

Helpful Hint

Did you know you can combine dish detergent with vinegar to create a super-cleaning agent? Great for cleaning tile in bathrooms….

18 of 26

5-B) Basic Functions to Know: Merge

Function

For combining data sets

Three different kinds of merges – 1:1; 1:m/m:1, or m:m (almost never right to use this last one)

The first numeral or letter corresponds to the “master” data, the second to the “using” data

Can use just one or multiple identifiers

merge 1:m person_id quarter using “data set name here”

Always returns an _merge variable

Depending upon the purpose of the merge, you may not want to see a value of 1, 2, or 3 on this variable

Helpful Hint

Always double check the _merge variable after a merge and identify a validation observation beforehand from both data sets so it is easier to check how things went wrong (if they did!)
Can suppress _merge variable – don’t unless you really know what you are doing!

19 of 26

5-C) Basic Functions to Know: Global and Local

Function

Allow you to create temporary or permanent macros

Varied – local macros are helpful to store temporary values used in other calculations or loops, I personally use global macros to streamline my code so that I am not typing the same variable names over and over
Local macros may be referred to like so:

local tempvar=1
di `tempvar’

Global macros may be referred to like so:

global tempvar 1
di $tempvar

Helpful Hint

Multiple “for” loops in Stata can get kind of tricky, and Stata may not treat them properly, either (anecdotal evidence) so replacing a “forval” with a local macro can be a good solution
Use global macros and loops together! Need higher order or logged terms for a bunch of continuous variables? Create a global macro and just reference that!

20 of 26

5-D) Basic Functions to Know: Foreach and Forvalues

Function

Allow you to loop code across many variables or values (among other things)

If you need to recode numerical missing value to Stata missing values across a number of variables, foreach is very helpful

foreach “var” in varnames {

recode `var’ (-9999 = .)
}

Or, let’s say you want to rename a series of numbered variables:

forvalues num=1/5 {

rename var`num’ newvar`num’
}

Helpful Hint

Use these as often as you can but when you first begin using them, always make sure to double check that they worked as intended

21 of 26

5-E) Basic Functions to Know: While

Function

Pretty versatile, I use it for controlling looping functions but can also be used when the “by” command cannot be used (as is true in some egen functions)

Here’s how I have used it in the past within a foreach loop

local i=99
while `i’>=1 {

foreach var in varnames {

replace `var’=`i’ if varname<=`i’
}
local i=`i’-1
}

Helpful Hint

Useful for when you need to use multiple loops, and especially when forvalues misbehaves in the loop or you just cannot figure out how to get it to work.

22 of 26

5-F) Basic Functions to Know: Statsby

Function

Allows you to create separate data sets from results stored in Stata memory

I’ve used it to extract results out of Stata just about in their entirety to then export them over to Excel.
Let’s say I suspect that my results would likely be different in male/female subsamples – I can estimate and save these results to another data set.

statsby _b _se, saving(“data set name here”, replace) by(male): regress depvar indepvars
This will take all coefficients (_b) and standard errors (_se) from both models and store them in a new data set.

Helpful Hint

This can be pretty useful for obtaining summary statistics across subsamples in your data, especially if you have many of them. Would save a lot of copy and pasting on your part if you could just export the results right into Excel.

23 of 26

6) Data Exercises with Advanced Functions

Or, the simple nature of things that should,

by all means, be harder than they are.

24 of 26

7) Some Other Functions to Learn

This is a list not just for you, but for myself also (hence why I don’t go over them at all)

Else

Helpful in loop commands

Outreg/Outreg2

Creates output for you, almost like LaTeX
Can set up multiple formats for tables that work for different journals

Project

User-written command that allows you to link multiple do-files and data sets within a particular named project you are working on
Appears to be very helpful with file pathing issues

Program

The basic foundation of user written programs – this function allows you to write your own Stata programs and test them out

1 of 26

2 of 26