Do men earn more than women?¶
It is often reported that women earn more than men. The Ontario government (in Canada) reports the salary of all public-sector employees making more than a given amount. This data is freely available.
dataset = pd.read_csv("fr-2017-pssd-compendium-20180320-utf8.csv")
dataset.head(5)
How many people do we have? About 130 thousands.
dataset.shape
It looks good but the file contains numbers as "$113,847.58". It is not recognized by Python as a number. So we need to remove the '$' and the ',' before we can parse it as a number. So we call the "replace" function.
dataset["salaire"]=dataset['Traitement versé'].replace( '[\$,]','', regex=True ).astype(float) + dataset['Avantages imposables'].replace( '[\$,]','', regex=True ).astype(float)
pd.options.display.float_format = '${:,.2f}'.format
salaireparposte=dataset.groupby("Poste").agg({'salaire':[np.size,np.mean]}).reset_index()
salaireparposte=salaireparposte[salaireparposte[("salaire","size")]>200].sort_values(("salaire","mean"),ascending=False)
salaireparposte.columns = [col[0]+col[1] for col in salaireparposte.columns]
salaireparposte[["Poste","salairemean"]].set_index('Poste').head(10).plot.bar()
dataset.head(5)
salaireparposte=dataset.groupby("Poste").agg({'salaire':[np.size,np.max]}).reset_index()
salaireparposte=salaireparposte[salaireparposte[("salaire","size")]>200].sort_values(("salaire","amax"),ascending=False)
salaireparposte.columns = [col[0]+col[1] for col in salaireparposte.columns]
salaireparposte[["Poste","salaireamax"]].set_index('Poste').head(10).plot.bar()
profsalaire = dataset[dataset['Poste'].str.contains("Prof") & ~(dataset['Poste'].str.contains("Dean"))]
profsalaire.sort_values("salaire",ascending=False)[["Nom de famille","Employeur","salaire"]].head(100)
Ok. So we have the dataset, but there is no gender identification. Thankfully, there are available statistical models that allow us to categorize first names.
genderstat = pd.read_csv("us-likelihood-of-gender-by-name-in-2014.csv")
genderstat[["sex","name"]].head()
But first, let us check that things work as they should:
from unidecode import unidecode
def gender_first_name(name):
try:
return genderstat[genderstat['name'].map(lambda x : unidecode(x))==unidecode(name)][["sex"]].values[0][0]
except:
return "UNKNOWN"
for n in ["Nathalie", "Julie", "Pierre", "Yves", "Daniel", "Lohan"]:
print(n, gender_first_name(n))
Ok. So now we have two tables. What are we going to do to put the data back together? One solution is to use a "join".
We have a 'name' column in one dataset, and now we will create another name column on the other data set, after making sure to trim out accents so that "Éric" becomes "Eric".
from unidecode import unidecode
dataset['name'] = dataset['Prénom'].apply(lambda n : unidecode(n) )
dataset[['name','Prénom']].head(10)
datasetwithgender=pd.merge(dataset,genderstat,on="name")
datasetwithgender.groupby("sex").size()
datasetwithgender[datasetwithgender["Poste"]=="Physician"].groupby(["sex"]).agg({'salaire':[np.mean,np.max,np.median]}).plot.bar()
datasetwithgender.groupby("sex").agg({'salaire':np.mean})
datasetwithgender.groupby("sex").agg({'salaire':np.max})
waterlooprofessor = datasetwithgender[(datasetwithgender["Employeur"]=="University of Waterloo")
&(datasetwithgender["Poste"].str.contains("Professor"))]
waterlooprofessor[waterlooprofessor["sex"]=="M"]["salaire"].plot.hist( bins=20)
waterlooprofessor[waterlooprofessor["sex"]=="F"]["salaire"].plot.hist( bins=20)
waterlooprofessor.groupby("sex").agg({'salaire':np.max}).plot.bar()
waterlooprofessor.groupby("sex").agg({'name':np.size}).plot.bar()
ottawaprofessor = datasetwithgender[(datasetwithgender["Employeur"]=="University of Ottawa")
&(datasetwithgender["Poste"].str.contains("Professeur"))]
ottawaprofessor[['name','sex',"salaire"]].head(5)
ottawaprofessor.groupby("sex").agg({'salaire':np.mean})
ottawaprofessor.groupby("sex").agg({'salaire':np.max})
ottawaprofessor[ottawaprofessor["sex"]=="M"]["salaire"].plot.hist( bins=20)
ottawaprofessor[ottawaprofessor["sex"]=="F"]["salaire"].plot.hist( bins=20)
