Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women (reuters.com) 381
Jeffrey Dastin, reporting for Reuters: Amazon's machine-learning specialists uncovered a big problem: their new recruiting engine did not like women. The team had been building computer programs since 2014 to review job applicants' resumes with the aim of mechanizing the search for top talent, five people familiar with the effort told Reuters. Automation has been key to Amazon's e-commerce dominance, be it inside warehouses or driving pricing decisions. The company's experimental hiring tool used artificial intelligence to give job candidates scores ranging from one to five stars -- much like shoppers rate products on Amazon, some of the people said. "Everyone wanted this holy grail," one of the people said. "They literally wanted it to be an engine where I'm going to give you 100 resumes, it will spit out the top five, and we'll hire those." But by 2015, the company realized its new system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. That is because Amazon's computer models were trained to vet applicants by observing patterns in resumes submitted to the company over a 10-year period. Most came from men, a reflection of male dominance across the tech industry.
[...] Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.
[...] Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said. The Seattle company ultimately disbanded the team by the start of last year because executives lost hope for the project, according to the people, who spoke on condition of anonymity.
AI really can't replace everything. (Score:3)
Re: (Score:2, Insightful)
Well, there is a huge imbalance between genders in IT, due to different reasons (which I don't want to speculate about, James Damore tried and we all know what happened). So the system must have a bit of positive bias towards women to correct for this - to help restoring balance. However, the cold heartless AI can't factor in this ... what data you feed in defines what model you get. So, is the data biased?
Re:AI really can't replace everything. (Score:5, Interesting)
there's a huge gender imbalance in nursing and primary education as well; when will society get around to 'fixing' that?
Re: (Score:3)
there's a huge gender imbalance in nursing and primary education as well; when will society get around to 'fixing' that?
There are efforts to recruit more men into these professions. This is especially important in early primary education where there is evidence that boys, and especially black boys lacking male role models in their lives, do better with male teachers. Over feminization of education is not a good thing for our society. We need more men teaching kindergarten.
Re: (Score:2)
You first!
Re: (Score:2)
Re:AI really can't replace everything. (Score:5, Insightful)
Yep, and you thought that #MeToo being weaponized was bad. You *might* get away from that label with false accusation....maybe/
But if anyone every accuses you of a child related crime, again, even if FALSE....good luck on having a life after that.
Guilty until proven innocent...and even then, well, you know....can't trust you.
Re:AI really can't replace everything. (Score:5, Insightful)
The thing is, most men are not going to make advances on a teaching assistant, or seek to ruin that person's career.
All men are however at risk of a false accusation.
Teaching assistants can make their own choice of which risks to their career they accept. The person to whom you replied is doing exactly the same, by eliminating an obvious risk vector.
Re: (Score:3, Insightful)
They are being fixed. There are programs by nurse's unions on hiring more male nurses, and there are programs for teachers as well in increasing the number of male teachers teaching elementary school.
Don't assume that because you don't know about it it's not a big deal. There are programs for increasing the proportion of females in trades (construction, welding, etc) run by various trades
Re: (Score:2)
Re: (Score:2)
what data you feed in defines what model you get.
No it doesn't. This is trivial to fix. You just slap a softmax layer onto the output of your NN to correct the bias.
So if your candidate pool is 70% men and 30% women, and you want your output to reflect that, then instead of just accepting the gender-agnostic "best", your softmax layer shapes the output so that you pick the top 70% of men and the top 30% of women ... or any other statistical distribution that you want.
So instead of "scrapping" their recruiting tool, Amazon could have easily fixed it with
Re: (Score:2)
If you have 70% men (e.g. 70 men) and 30% women (e.g. 30 women) - then pick the top 70% of the men and top 30% of the women: 1 you probably still have way too many people (58 hires), and if you took them all you would end up with ~84% men to ~16% women.
Re: (Score:3)
it is still much easier for a mediocre man to get hired than a mediocre woman
That is just total bullshit.
There just happen to be a lot more mediocre men trying to get hired.
Re: (Score:2)
As hard as you want to say, sometimes you still need an actual person doing the job. That person will be biased in some way other another too, so I guess it's not a perfect system any way you look at it.
Amazon goes to extremes though for the hiring process for any professional job: the interview loop must include someone not associated with the hiring team, and that person runs the interview process (and gets lots of extra training and auditing). They're going to great lengths to avoid "we're desperate to hire anyone, so we'll take someone almost good enough", but the side effect is strict objectivity.
But the law is pretty clear that you can't use any protected category in your resume sorting even if it's
Re: (Score:2)
But the law is pretty clear that you can't use any protected category in your resume sorting even if it's statistically predictive,
No, the law is NOT clear on this.
Many companies have been fined by the EEOC, not because their hiring process was shown to be biased, but because the outcome of the hiring process did not match the race and gender profile of the candidate pool. These were, of course, big companies with workforces big enough to statistically analyze. Smaller companies can get away with almost anything short of overt discrimination.
So while the law does not explicitly require quotas, and even seems to prohibit them, in prac
Re: (Score:2)
Many companies have been fined by the EEOC, not because their hiring process was shown to be biased, but because the outcome of the hiring process did not match the race and gender profile of the candidate pool.
Pics or it didn't happen.
Re: (Score:2)
You need to distinguish between "deciding who to interview" and "deciding who to hire". There's no real defense on the former: you had better be interviewing at least as many people in each protected class, proportionally, as you have in your candidate pool. You can defend yourself on the latter, but you'd better be prepared to show that each and every interview was decided on objective criteria that don't include protected class. But you have a much bigger risk of a lawsuit than an EEOC fine on the la
Re: (Score:2)
Is this news? (Score:5, Insightful)
Train algorithm with data in hand, algorithm's output mirrors data provided. They can't possibly be shocked by this, can they?
Re: (Score:2)
Re: (Score:2)
Because most people have been convinced that AI can turn garbage inputs into perfect outputs.
On two occasions I have been asked, — "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" In one case a member of the Upper, and in the other a member of the Lower, House put this question. I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
/obligatory Babbage quote
Re: (Score:2)
Didn't they have to tell Watson to forget everything you learned from Urban Dictionary?
Yes, they did. [theatlantic.com] They couldn't even feed Wikipedia articles in without some editing.
Re: (Score:2)
Nothing good comes from visiting the Urban dictionary.
Before hitting that link, drink yourself just short of blackout drunk and pour a tumbler of liquor. So you can 'unread'.
Alternatively, 'blow the head'.
You have to do it before the thing read gets committed to long term memory.
Re: (Score:2)
Train algorithm with data in hand, algorithm's output mirrors data provided. They can't possibly be shocked by this, can they?
I recently began working with AI, and was surprised at how simple the algorithms are. I think a lot of people expect them to be extremely complex.
Training the AI is where it gets complex. This is where shaving a couple percent off computation times can really make a difference. Finding good training data. Selecting the right optimization algorithm. Waiting for the system to learn. That's what's difficult.
Re:Is this news? (Score:5, Insightful)
The problem it isn't Empirical data, It is incomplete data.
Right now If I were to look at my desk, and feed it to an AI. All Electronic Devices are Made with Black Plastic, and Coffee vessels are Navy Blue. If I expose it to an beige keyboard that is in the storeroom. It would say that it isn't an Electronic Device. If I Took out my White Mug, from the shelf, it wouldn't think I could use it for coffee.
Incomplete data, isn't Empirical Data.
Re:Is this news? (Score:4, Informative)
Because the algorithm finds correlates to explain the data, and it always takes the "path of least resistance".
For example, say that women weren't hired as often because women tend to lack the necessary job experience more often than men. Then, the algorithm is looking for the *simplest possible* variable that correlates with that decision, so it notices otherwise irrelevant terms like "women's" or that successful candidates are more likely to use "masculine words". Those female candidates weren't rejected *because* of those word-selections, but because they tended to have less experience, but the algorithm isn't smart enough to understand that, so it glomps on b.s. word choices as a deciding factor.
For example, say that more successful candidates were more likely to wear good shoe brands, so your algorithm decides that choice of shoes should be a factor in hiring, because it's just that much easier for the algorithm to pick that signal out of the noise of the data you've fed into it. The original hirers didn't actually give a toss what shoes you wore, it's just that more successful people could afford better shoes to start with, but the algorithm hones in on "only hire nice-shoe-people" as if it's important. This is one of the problems with machine learning. It learns the "what" and not the "why" behind data correlations.
Re: (Score:2)
Wouldn't work that way unless the men had better 'outcome metrics'. Simply having more of them wouldn't make their traits get a positive score.
Sounds like the algorithm found proxies for sex, but it _wouldn't_ have done that if the women had equal 'outcome metrics'.
They likely have a problem with small datasets. For example, the AI assigned a couple of 'women's colleges' a negative score, likely because they had hired an airthief or two from those schools. But you have to accept the possibility that th
Re:Is this news? (Score:4, Funny)
Guinness sells beer. Guinness does not sell wine. This is empirical data. Is it incomplete?
Yes.
Diago, the owner of Guinness, also owns a large percentage of Moet Hennesy.
Re: Is this news? (Score:2, Insightful)
No reason is sensible to you because you have already prejudged all reasons to be non sensible.
Re: Is this news? (Score:5, Insightful)
No reason is sensible to you because you have already prejudged all reasons to be non sensible.
Empirical data isn't racist or sexist...unless you're a liberal. Then if you think the data might hurt someone's feelings or make someone feel less special it's clearly [insert -ist/-phobe buzzword of the moment] and so is the person that brought up the imperical data.
I'll probably get downvoted for this, but you all know that it happens ALL THE TIME.
Re: (Score:2)
people don't complain about the data itself, it's usually the collection methods, how its interpreted, or its application that gets criticized
Re: (Score:3)
Re: (Score:2)
Narrow your focus. There is an argument about the language part of the SAT.
The math part _can't_ have a cultural bias.
Re: (Score:2)
Math is potentially much less culturally biased. Math as it exists is written in a specific unusual language, which is definitely not even "mainstream English-speaking American culture".
Asserting that culture and language cannot matter is very obviously wrong.
If I plunked you down in an advanced math course in Beijing along with Chinese of similar academic background to you, do you believe you would perform nearly as well as the locals? Why not? Think language and culture just might contribute to your di
Re: (Score:2)
Math is written in 'math'.
Which isn't anybody's native language.
I'd do a whole lot better in a Chinese math course than a Chinese lit course, but that's not a fair comparison to English speakers of various backgrounds taking the SATs.
Math is biased against irrational cultures, I'll give you that. SJWs are generally bad at math.
Re: (Score:3)
From your bold assertion upthread: 'Math as it exists is written in a specific unusual language, which is definitely not even "mainstream English-speaking American culture".' That language is called 'math'.
That's a long way from being able to say 'culturally biased', which IS a big old stretch, even for the language part of the SAT.
Math testing is not culturally biased, except against 'feeling culture'. Which IT SHOULD be biased against.
Re: Is this news? (Score:2)
I already perfectly explained why you cannot be presented with information that will change your mind. You are a racist. No handwaving involved.
Re: (Score:2)
They removed the French and became self-governing with a "made" nation to start with. It quickly fell apart into the hellhole it is today. This is what you would expect and is by no means an anomaly.
That Haiti failed is not a mystery. Any new nation under those circumstances would be likely to fail. It might of been a success but France and its best prospective trading partner very much wanted it to fail.
But you are not interested enough about the topic to investigate their history at all, just looking at the color of their skin is all you need to know, apparently.
Let's consider the example of Greece, literally the origin of what we assert is much of the secret sauce of Western awesomeness. Why is G
Re: (Score:2, Informative)
The real problem is the algorithm wasn't biased.
Ding ding ding! We have a winner!
Tell him what he's won, Johnny!
I had to scroll this far down to reach the actual answer?
Slashdot, I mourn for thee.
AC is exactly correct. The problem is not that the algorithm is racist or misogynist, it's that it's not biased at all while a large portion of our society *is* strongly biased and does not *want* unbiased answers that conflict with their current political/ideological/cultural/racial prejudices and stereotypes that are the tools essential to sustaining and growi
The law... (Score:5, Insightful)
When government reviews your hiring they expect you to show that your diversity level is consistent with the normal spread of minority groups ( some consideration of candidate pool MAY be given.)
In other words, if your only criteria is hiring whomever best for the job, you will likely be operating illegally and subject to fines and lawsuits. This is the product of laws that are designed to create social engineering based restrictions based on someones religious idea that any measurable discrepancy in minority placement must be corrected.
Re:The law... (Score:4, Insightful)
Also, if you are running your company with only the end goal in mind then you are probably doing something illegal. Laws exist to protect the interests of society, not the interests of individual companies or people.
Re: (Score:2, Funny)
Re: (Score:2)
FTFY.
Re: (Score:2)
This already happens.
Many companies do hire lots of foreign workers for way less money, a lot of jobs are being offshored regularly.
Many foreign workers are also employed locally especially for various low paid unskilled work, as they are willing to work for lower wages.
For higher skilled positions, especially local rather than offshored ones foreign workers are also frequently used, but they are less likely to accept lower wages in such conditions, and why would they? If they are equally skilled such that
Re: (Score:2)
In other words, if your only criteria is hiring whomever best for the job, you will likely be operating illegally and subject to fines and lawsuits.
Until you can demonstrate that women or non-whites are not physically capable of being "the best for the job", then this reasoning is flawed.
And if your "proof" is the lack of not-white-men in these positions, you're falling for the same incomplete data as the AI in TFS.
Re: (Score:2)
Until you can demonstrate that women or non-whites are not physically capable of being "the best for the job", then this reasoning is flawed.
And if your "proof" is the lack of not-white-men in these positions, you're falling for the same incomplete data as the AI in TFS.
It's not all-or-nothing, it's bell curves. If you allow race as a consideration for e.g. software development, you'll get a statistical model that tells you to mostly interview Asians. That's an accurate predictive factor (one of many) of how well a candidate is likely to succeed at the job. It's also illegal (well, not the discrimination against whites of course, but against all the non-pariah races).
It's not about "incomplete data as the AI" unless the coders were complete idiots. Amazon has a huge po
Re: (Score:2)
It's not all-or-nothing, it's bell curves.
Yes, you have to establish that such bell curves differ based on gender and not-whiteness. And again, if you're just looking at outcome as your proof, you're falling for the same incomplete data as the AI in TFS.
It's not about "incomplete data as the AI" unless the coders were complete idiots. Amazon has a huge population of developers.
That huge population was created with bias in mind, not only in initial hiring but in how women are treated in the workforce over the years. Thus you can not trust that this data is completely impartial.
It's the nature of machine learning systems that you can't really prove what criteria they've "learned" from your training data
Sure you can. The machine learning system will produce a result that closely matches the train
Re: (Score:2)
Yes, you have to establish that such bell curves differ based on gender and not-whiteness. And again, if you're just looking at outcome as your proof, you're falling for the same incomplete data as the AI in TFS.
The point is, it doesn't matter how you establish it. All the details you go into are irrelevant. The people you interview can't be less representative than the candidate pool of protected classes, regardless of why - even if you had ironclad proof, doesn't matter.
It's the nature of machine learning systems that you can't really prove what criteria they've "learned" from your training data
Sure you can. The machine learning system will produce a result that closely matches the training data. When your training data is the result of bias, the machine learning system will strive to continue that bias.
Man, you really love to say "bias". But you've missed the point here:
* You cannot, from the output of a machine learning system, prove that it isn't using X as criteria in screening candidates.
* You might need to be able to prove that legally.
*
Re: (Score:2)
In other words, if your only criteria is hiring whomever best for the job, you will likely be operating illegally and subject to fines and lawsuits. This is the product of laws that are designed to create social engineering based restrictions based on someones religious idea that any measurable discrepancy in minority placement must be corrected.
That's not quite true. The rules for who you interview are different from the rules on who you hire.
What you say is true for the former: if the people you interview don't at least match the candidate pool for any protected class, you're boned regardless of why. It's a bit different for hiring, where you can defend yourself by showing that your process is objective.
In practice, the government tend to focus a lot on who gets interviewed, as that's strictly numbers and easy to audit and enforce. The fear fo
Re: (Score:2)
Bingo! At a former employer, they went out of their way to increase the diversity of the recruiting pool ... to no effect. Mostly because while the Fed regulator was screaming about the 10% of the local population that was a 'minority ethnic group', they were ignoring the detail that group was less than 1% of the graduates from computing courses at the local colleges ... a number that was reflected in the makeup of the IT department.
To be fair to the company, they broke 0.5% minority hiring, while the gra
Re: (Score:2)
When they review you for granting a government contract.
Don't do business with government and they need more evidence than just statistics.
The data set was flawed (Score:5, Insightful)
Amazon trained their AI using the dataset that reflected their business practices as they currently are (flaws and all) but what they wanted was a data set for the practices they wanted to become (i.e. the ideal).
Finding a training dataset that reflects the ideal is going to be extremely difficult, particularly in an area where that ideal is so poorly defined.
Re: (Score:3, Insightful)
They just have to train it with data that shows only success from women and failure from men.
Easy: Just make it against the rules to review women employees with anything other than 'perfect'.
Their dataset might be 'just fine', could be their assumptions and goals are broken.
The reason businesses love "AI".. (Score:2, Interesting)
It lets them make immoral business decisions but not be personally held accountable for them.
Facebook shows real estate ads only to white professionals. Amazon only hires male chinese engineers. Google endlessly manipulates its search for political reasons.
But when questions get asked, it's always that pesky old AI that did it!
Get used to it.
It's not "bias" if it just reports the facts (Score:4, Insightful)
Bias is a non-factual prejudice against someone. That is why it is considered unfair. If the facts are that 80% of the population of people who do the work you want are named "Dave", then it is not a sign of a moral failing if your AI exhibits a strong preference for another Dave.
Re: (Score:2)
I wouldn't call it a moral failing if unintended, just bad design.
Re: (Score:2)
Re: (Score:2)
but the name itself has nothing to do with suitability for the position
I think the OP meant that the AI was preferring people named Dave, even though it didn't know people's names. That is analogous to what was happening with the AI filter: It didn't know people's sex, yet it was still picking more men than women.
Re: (Score:2)
AI systems are all about Biases, it finds a pattern and gives it a number. A we as humans have Biases because our brains find a pattern and qualify it. Patterns are not facts.
On average Men are physically stronger then women this can be quantified and measured. However there are a LOT of women who are stronger then the average men, They are also a lot of Men who are weaker then then the Average Women.
So if a job requires physical strength, you may get on average more men who can do the work. But this sho
Re: (Score:2)
However there are a LOT of women who are stronger then the average men, They are also a lot of Men who are weaker then then the Average Women.
I don't think you understand how averages work.
Re: (Score:2)
You should either stop making up word definitions, or start using a better dictionary.
Re: (Score:2)
You should either stop making up word definitions, or start using a better dictionary.
Better dictionaries could help... Google dictionary comes up with the basic definition the GP quoted. Having worked with aviation sensors I tend to think of bias more as it is defined for sensors:
"If the output signal differs from the correct value by a constant, the sensor has an offset error or bias. This is an error in the y-intercept of a linear transfer function." https://en.wikipedia.org/wiki/... [wikipedia.org]
In this case, and in the case of a lot of computer programs, bias can be generated that has nothing to do
Re: (Score:2)
Bias is a non-factual prejudice against someone.
Floating point numbers are prejudiced against someone? [wikipedia.org]
New Technology, Same Old Problems (Score:5, Insightful)
If the training data has bias, then the AI will learn to have that bias.
The trick is developing training data that doesn't reflect the biases of the humans that performed the task in the past.
Re: (Score:2)
Well, you could simply use the performance of people, instead of the judgement of said performance by others. Of course, that would require you to introduce proper metrics. Which is something that a lot of managers resist in this field as soon as it applies to "thinking jobs". I wonder if the fact that the group includes their own jobs has any influence on that distaste.
Re: (Score:2)
I do not doubt that loss of power is a factor, but there is a very real question of whether the business is willing to make the investments and stay the course long enough that the end result is worth the trouble. Skepticism here can be very rational.
"Proper metrics" that cannot be gamed are very expensive. It is not just the rubrics, but building a culture that works with the rubrics. If you think "proper metrics" will simply work by virtue of their awesomeness, you are definitely doomed.
Re: (Score:3)
Well, you could simply use the performance of people, instead of the judgement of said performance by others. Of course, that would require you to introduce proper metrics. Which is something that a lot of managers resist in this field as soon as it applies to "thinking jobs". I wonder if the fact that the group includes their own jobs has any influence on that distaste.
Would you like to share some of your proper metrics? Lots of (all?) companies use bad metrics, but is there any other kind?
AI isn't (Score:2)
We have good expert systems that can do amazing things with ultra controlled inputs.
Pretending computers have a bias vs anything is really dumb, articles and submissions like this are for controlling the narrative and keeping people poorly informed. The general populace is much smarter than they are given credit for, especially when they have the right information.
People that push this kind of nonsense ought to be ashamed. Slashdot used to be about the cool tech, anyone can go to vice(or pick your polit
Re: (Score:2)
Pretending computers have a bias vs anything is really dumb
No, what's dumb is pretending the computer is doing anything beyond its training data.
You give the computer biased training data, you get biased results. Not because the computer has bias, but because your training data is the result of bias.
Equality of outcome, an impossible fantasy (Score:2, Insightful)
If the results are biased, the data is biased and the process is biased, maybe the bias is normal?
No it didn't... (Score:2)
Ahh more slanted news to push an agenda (Score:5, Interesting)
When you read this article it doesn't say anything about this algorithm not 'liking woman'. Based on the parameters it was given it chose to rank candidates based upon the factors it was trained to look for. It's also somewhat telling how the writers of this tripe chose to specifically highlight how the algorithm chose to downgraded candidates from two all female colleges without saying why they were downgraded. As if the fact that it's an all female school is more important than the quality of the candidates that came out of the school.
At the end of the day this bullshit is more about how the media writes headlights to illicit emotional reactions instead of reporting the hows and the whys of a situation. And on that note I'd like to see someone actually start writing algorithms to to replace tech reporters so we can get ride of garbage tier activist journalism like this article.
Similar To The James Damore Ideological Attacks (Score:4, Insightful)
Yes.
Both this ridiculous garbage reporting and the apoplectic shitshow from ideologues in the press over James Damore's memo are not just the usual bland claims of sexism.
There are long known and well researched gender differences in interest preference going all the way back to infants - long before any possible way to for the results to be explained by 'societal sexism' or other such nonsense.
Feminist dogma is 100 percent counterfactual this basic and well researched science.
Hence why the over the top attacks on anyone and anything that brings to light these fundamental differences in the abilities of men vs women in technological jobs.
The reason there is such a huge disparity in male hires in tech companies is a direct result of those well established gender differences. The candidates being selected are at the very, very top end of the bell curve in both intelligence(where men have a significant advantage) and a lifetime of interest in and drive compared to female applicants in general.
Of course the usual 'argument' and response anyone pointing these basic facts out is screetching that the claim is women aren't as capable as men.
Any individual woman can be just as capable as a man in tech.
However, that is not true at the population level where men will significantly outpace women in the number of highly qualified candidates.
What does it do if you remove all gender? (Score:5, Insightful)
Purge any submission to the system of a gender identifier... women's or men's anything... remove names in case that is factored... literally provide nothing in the submission that would definitively define a gender.
Then see what it does.
My experience with these systems is that they don't actually factor gender but that the end result of is that there is a gender imbalance.
However, if there is an imbalance and the system was given no indication as to gender then there is no gender bias.
You can't cite persecution or preference if the system can't even know. And generally these fairly common and consistent imbalances are made without reference to gender itself.
Generally it is factoring on other criteria that give the same result but which are not gender. Work experience is a big one... breadth of skill set is another.
And if you took the total population and look at which portion of the population had that work experience and breadth of knowledge, you'd find it more closely matched the hiring patterns of these systems. Which means it isn't factoring on gender.
Now... this is assumption to some extent on my part. I've audited these systems in the past and what I am describing above is the pattern I've seen.
As to what the Amazon system was doing... I'd have to audit it.
What I'd probably try is a word replacement/purge of all terms that would signify gender or I'd just change a bunch of rejected female resumes to say they were male and see if they got accepted and vice versa.
If the system actually changed its decision based on gender then that's a smoking gun that it is doing things on the basis of gender.
But I'd find that very surprising.
Machine learning is unpredictable so I'm hardly going to claim to know what the damned thing was doing. For that reason I wouldn't actually use machine learning in this application. I'd use a very clear rules based system where everything it was doing was known to the programmers.
Those systems are completely fine for this sort of work and you can very easily audit the code for them.
The best way to deal with this is to first be gender blind. You literally do not factor for gender at all.
That will give you an imbalance probably... you can make as many diversity hires as you need to after that. But your core hiring pool should be merit based unless you want to go out of business.
Re:What does it do if you remove all gender? (Score:4, Insightful)
Purge any submission to the system of a gender identifier... women's or men's anything... remove names in case that is factored... literally provide nothing in the submission that would definitively define a gender.
Writing style can tend to be different between genders. And you can't really remove that without feeding it blank pages.
Re: (Score:2)
I'd just change a bunch of rejected female resumes to say they were male and see if they got accepted and vice versa.
This exactly. Do it both ways. Make some male profiles female and some female profiles male and check what happens.
Then identify other gender-typical features, one by one. Gaps in the CV (pointing to child raising times), remove any written text (gender styles of writing), etc.
This could be a really cool toy to figure out some of the recruiting prejudices and understand what the gender gap actually is. Because we already know it is not just gender. It is also experience and other life choices that are indir
Re: (Score:2)
Purge any submission to the system of a gender identifier...
It is not that easy. I assume they run some kind of pattern matching algorithm and it can just as easily focus on non-obvious gender differences as obvious ones. It could be that hobbies or volunteer work are just as good predictors of gender as a name (and therefore sources of bias). Maybe even the choice of words or textual layout could make a difference. It would take a lot of scrubbing and perhaps a complete rewrite of the job applications to remove all traces of personality.
When they say the result is
Re: (Score:2)
Purge any submission to the system of a gender identifier... women's or men's anything... remove names in case that is factored... literally provide nothing in the submission that would definitively define a gender.
Then see what it does.
Except there are plenty of things that are very germaine to a resume that show gender but cannot be filtered out without altering the resume. For example, if education has Mount Holyoke, Spelman College, even something like International University of Technology for Women, your algorithm will know it's a woman's resume. Or maybe they went to a coed school but were a president or founding member of their school's Women Coders club? Maybe they had a successful career but took 2 years off when they got preg
Re: (Score:2)
Purge any submission to the system of a gender identifier... women's or men's anything... remove names in case that is factored... literally provide nothing in the submission that would definitively define a gender.
Then see what it does.
Another interesting experiment would be to leave the gender identifiers in the training data, but make sure it's carefully balanced. Then test it by stacking it so that the only qualified candidates are all one gender, then see if the computer would hire unqualified candidates just to provide a politically correct result.
How can smart people be so dumb? (Score:2)
This comes up frequently in high-tech companies: If only we could automate decision-making without involving people! Imagine!
This is literally the dumbest thing you could do, right up there with "B people hire C people." As an interviewer, I always looked at resumes to guide my interview approach, but in most cases it was impossible to make any decisions based on a resume. Even if you assume that the person didn't outright lie, you're looking at 4-line summary of 3-year work periods written by a writer w
The bigger news (Score:2)
Pre-process your training data (Score:2)
Pre-process the training data so that exactly half of the input is from male applicants, and half from female. If there are more male entries in the complete dataset, then randomly remove them until it's exactly 50:50. Yes, this means tossing out potentially valuable information. However, if male applicants make it to the top 5, then they could be further compared against each other using the full male dataset. Surely if I thought of this in 5 sec they could too?
If the problem is that not enough women are i
Re: (Score:2)
"bias" (Score:2)
Well, the last thing any of our new diversity-obsessed saviors want is to (openly) specify hiring criteria, and generally computers require you to specify things. So it's not surprising that we run into these little snafus.
That said, I would have thought that "AI" would be a, er, godess-send for these folks ... just train it for awhile, and nobody will have any way to prove why it makes the decisions that it does. Sounds perfect for "diversity" hiring.
Amazon sucks at programming (Score:2)
Simply reading the instructions the AI was apparently using, according to the article, tells me that whoever created this AI was either (1) a moron or (2) a bigot.
Re: (Score:2)
People do not program in particular words/phrases to bias on.
The system automatically compares all/many aspects of the document (e.g. all words and short word sequences) against the same aspects of resumes of "hired and successful" past applicants to the company.
The system learns statistically which words/phrases/aspects of the documents correlate with "hired and successful".
Programmers don't enter into this at all. It's general l
Balancing your dataset is basic. Not the problem. (Score:5, Insightful)
The claim that "the industry is dominated by men and therefore we couldn't train this in a gender-neutral way" is totally bogus from a machine-learning perspective. All that is needed to eliminate a bias arising from dataset imbalance is to balance the dataset.
More likely they realised that when using dispassionate criteria for optimal hiring, it would become very likely they'd not get the desired "Women > Men" politically correct outcome for all sorts of statistically valid reasons, and figured such optimal hiring was not worth its salt against all the money lost from lawsuits and bad PR in a time of a politically tense climate favouring women.
I completely agree with their choice, and would do the same. No need to feed oil to the fire
Well... (Score:3)
The A.I. doesn't care about being politically correct.
Maybe the A.I. has computed something we're not aware of.
Unfortunately, people will force political correctness into the A.I. and we'll never learn the truth. /sarcasm (or is it?)
Typical management (Score:2)
Make 500 models, teach it to recognize some 50,000 terms.
AI does HR's job too well.
Executives kill project.
Obligatory smbc (Score:3)
https://www.smbc-comics.com/co... [smbc-comics.com]
job application software sucks and is easy to Bias (Score:2)
job application software sucks and is easy to be Bias on any group that you want.
also they want way to much info up front.
Re: (Score:2)
No that is an incorrect statement.
The Algorithm finding that more men then women are being hired. Decides that gender is a factor to be considered. It is weighing factors that really do not matter.
Re: (Score:2)
That interpretation is very silly. They're trying to figure out "as a percentage of candidates with resumes like X that we interview, how many do well at the job", and make all such statistic correlations that they can.
It's not about "they have fewer women on staff", obviously. It's about "how do female candidates fare". Amazon, like all of the Big 5, is trying to address gender balance, and so will interview women with less chance of getting hired than their male candidates. They want more women hired,
Re: (Score:2)
Do you really think Amazon hired a statistically significant number of 'Vassar girls'?
The one they did, likely didn't make it out of probation. Hence the 'AI' gave that trait a negative score. It could be right.
Re: (Score:2)
just remove the gender part of the application form, or otherwise obscure it so the AI doesn't factor it in. Surely they've thought of this?
There are other parts of the application that can be used to easily infer gender.
This has bee the same problem with race as well. For example, your address has a very indicator of your race.
So one solution (Score:2)
Bounce the incoming resumes against both models, and find the good matches according to some threshold.
Now you are free to tweak the ratio of candidates coming from both automated selecti
Re: (Score:2)
I can imagine the conversation at Amazon...
"Goddamnit! The top CV's picked from this impartial unbiased machine learning algorithm are all men! It must be discriminating against women somehow, even though we aren't including gender in any of the data. Tweak it until more women pop out, or we're fucked once this gets out on Twitter..."
Maybe it *was* picking the best ones... and men just make better software engineers?
Yeah, yeah, sorry double plus ungood thoughtcrime. Warm up room 101...
Substitute women with H1B.
Top CVs picked are H1Bs. Double ungood though-felony!!
Re: (Score:2)
You would think a company called "Amazon" would show preference to women.
And they would only provide one service: death by snu snu!
Re: (Score:2)
Actually, the AI simply perpetuated whatever trends the humans were doing.
It makes no value judgement, it's just a statistical analysis saying how similar new thing is to samples that were declared 'good' before by humans. There's no room to declare the AI some higher unbiased authority.
Re: (Score:2)
Why is this downmod? That's what happened and was the issue and what they had to address.
Who is fantasizing this is an attack on women? That's disasterbation.