EmailRSS

Using Invoke-WebRequest to Find a Job

Posted on by

Every job site should have an alert system. As far as I can tell, the TeachGeorgia website does not have this feature. If you are wanting a certified position in a Georgia school system, this is a problem. When my wife asked if we could generate alerts, I said sure – let’s have a PowerShell dinner date. After all, a couple that scripts together stays together. While you may not get to enjoy the pizza that accompanied our PowerShell console, you can still learn how to use the Invoke-WebRequest cmdlet to navigate a website and to build your own tool.

The hardest part is finding what you need. We started by running a search in Chrome on the TeachGeorgia website for a single position type. For a media specialist position anywhere in Georgia, the URL for that search would be:

https://www.teachgeorgia.org/AdvSearch.aspx?Subjects=36&Systems=All

The question mark after /AdvSearch.aspx tells us that we have URL parameters that we can customize. Specifically, we can filter the Subjects and Systems (school systems) values. By playing around with the URL, we can see that multiples Subjects and Systems can be added. The TeachGeorgia website has a list of subjects and systems – we just now need to match the IDs to the names. This new URL may look something like:

https://www.teachgeorgia.org/AdvSearch.aspx?Subjects=12,36,72&Systems=6481,6631,7771

The next few steps are done with the developer tools in Google Chrome. The steps in other browsers are a bit different. Visit the address above and you will see that there is a table of jobs in the middle of the screen. Right click anywhere in that table and choose Inspect. The Chrome developer tools should open and should default to the elements tab. The spot where you inspected should be highlighted now. On the elements tab, move your mouse up a bit and watch as different parts of the page are highlighted. Find the line that highlights just the job listing table. It should look something like this:

In that line, you will see an ID. In this case, we see id=Body_gvJobList . This ID tells us exactly how to find this table when using Invoke-WebRequest!

Using Invoke-WebRequest to Filter Elements

Open PowerShell ISE and run the following:

$URL = "https://www.teachgeorgia.org/AdvSearch.aspx?Subjects=12,36,72&Systems=6481,6631,7771"

$TeachGeorgia = Invoke-WebRequest -Uri "$url" -SessionVariable "URISESSION"

Our first line creates and fills a URL variable. This is a modified version of that URL that we played with above. Our second line runs the Invoke-WebRequest cmdlet with our $URL variable along with a SessionVariable. The SessionVariable will allow us to maintain a single connection to the website. You can see a longer example of it here. In this script, the SessionVariable isn’t needed as we will only make one call to the website. It has been left in as an example though.

In the console pane of ISE, run $TeachGeorgia | Select * to view everything gathered in that web request. This view is a great way to get acquainted with the output of Invoke-WebRequest. We are interested in the AllElements listing. In the script pane below, run each line separately and observe the output. By the time that we are at the third line, we have filtered down to the exact table that we need. This will be the only line of the three that we keep.

$TeachGeorgia.AllElements 

$TeachGeorgia.AllElements | where id -EQ Body_gvJobList

$TeachGeorgia.AllElements | where id -EQ Body_gvJobList | select -ExpandProperty OuterHTML

Our final script can be seen below. At the top of the script, you will see a list of subjects and a list of school systems. I have included a few that you may be interested in. These numbers can be combined into the $Subjects and $Systems variable.

The next chunk ($user_agent to [System.Net.ServicePointManager]) are pulled from a generic Invoke-WebRequest template that I keep. After that, you will see the three lines discussed above. Finally, we just need a way to send that table as an alert. I used the Send-MailMessage cmdlet to accomplish this. You can see that using Invoke-WebRequest isn’t difficult – the difficult part is mapping what you need from the website.

#Computer Programming:   73
#Cyber Security:         104
#Earth Space Science:    13
#Information Technology: 72
#Media Specialist:       36
#Middle Grades Science:  65
#Other:                  41
#Science:                52
#Technology Education:   59

$Subjects = "Subjects=13,36,41,52,65"

#Brantley:   6131
#Camden:     6201
#GA Cyber:   78201201
#GA Virtual: 7771
#Glynn:      6631
#McIntosh:   6981
#Wayne:      7511
$Systems = "Systems=6131,6201,7771,6631,6981,7511,78201201"


$user_agent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0; MALNJS)"

add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

$URL = "https://www.teachgeorgia.org/AdvSearch.aspx?$Subjects&$Systems"

$TeachGeorgia = Invoke-WebRequest -Uri "$url" -SessionVariable "URISESSION" -UserAgent $user_agent
$OpenPositions = $TeachGeorgia.AllElements | where id -EQ Body_gvJobList | select -ExpandProperty OuterHTML

Send-MailMessage -to "ADDRESS" -From "ADDRESS" -subject "Open Positions"  -BodyAsHtml $OpenPositions -SmtpServer SERVER

Leave a Reply