Igor Kulman

In a few projects in the past I needed to do web scraping to get some data from websites that did not offer access via an API. I was using C# at the time and scraping web with Html Agility Pack was quite easy.

I now spend most of my time in macOS because of work projects so when I needed to do some web scraping again I did not want to install and set up Mono to do it again in C#. I decided to go with Swift, as I am now quite comfortable with the language after 4 years of using it daily.

SwiftSoup

The first thing I need to do was to found some library to parse HTML, some Swift equivalent to Html Agility Pack. I found SwiftSoup.

SwiftSoup allows you to access DOM in HTML documents and also HTML fragments. The usage is quite simple, you just need to know a thing or two about HTML.

Example

Let’s say you want to parse the Hacker News main page and scrap posts containing some specific keywords.

This is quite an artificial example but the idea is simple. You use the developer tools in your browser of choice to see the HTML of the parts of a website that you are interested in and try to get to them descending and filtering the DOM using SwiftSoup.

You first need to read the website and parse it

let content = try String(contentsOf: URL(string: "https://news.ycombinator.com/")!)
let doc: Document = try SwiftSoup.parse(content)

Looking at the HTML you can see it uses a table layout and all the posts are in a rows of a table with a class called itemlist.

How many times how you copied and pasted some code in your current codebase because there was no good way to abstract it? Maybe because it was some repeating code required by a framework or mapping of some data transfer structures.

Writing such boilerplate code is an error-prone waste of time, especially when there is a much better way: generating that code.

There are a few tools to help you do that, one of the most flexible of them being GYB.

What is GYB?

GYB is a lightweight templating system that allows you to use Python to generate text files. Those text files can be Swift source code files or anything else you need.

GYB is used in the Swift standard library codebase so it works well for generating Swift source code and it is a proven technology that works.

You can download GYB to your project from the Swift repository on Github

wget https://github.com/apple/swift/raw/master/utils/gyb
wget https://github.com/apple/swift/raw/master/utils/gyb.py
chmod +x gyb

I put it directly to the project directory and include it in the source control for simpler build and CI/CD setup.

Creating GYB templates

GYB templates for generating Swift source file look like Swift source files with some Python snippets for code generation.

This is probably best shown on an actual example. Let’s say you have a list of permissions that the app needs to support. Those permissions are then a part of a protocol and of a few structs.

Have you ever wondered how much time a day you spend waiting for Xcode to do your builds?

Working on a project that is almost entirely written in Swift I wonder that every day with a feeling that it is a non-trivial amount of time just wasted.

With Xcode build times you can actually measure it and know for sure.

Xcode build times

Xcode build times is a script that tracks all your daily builds and their duration.

You just set it up in Xcode to be called on build start, success and fail as stated in the project README.

It script is intended to be used as a plugin for BitBar. BitBar is a open source tool that allows you to show any script output in the macOS menu bar.

Just install BitBar and set the plugins directory to the directory with the Xcode build times script, as stated in the project README.

If you use a hackintosh you have to choose your hardware carefully to make sure it is supported by macOS. You can get Wi-Fi + Bluetooth card used by Apple as I did in my desktop, but sometimes you do not have much choice.

When I turned by old Thinkpad T440s into a hackintosh I bought a Wi-Fi dongle because the Intel AC7260 Wi-Fi + Bluetooth card is not supported by macOS, no Intel cards are.

Later I discovered and open-source project that aims to make Intel Wi-Fi and Bluetooth work on macOS and I was able to make the Intel AC7260 card work, no dongles needed.

Bluetooth driver

To get Intel Bluetooth working you need IntelBluetoothFirmware. It is a macOS kernel extension that that uses firmware binaries from Linux to make Bluetooth work.

Make sure your specific Intel card is supported, download the latest release and use the two kexts; IntelBluetoothFirmware.kext and IntelBluetoothInjector.kext. If you use Clover just copy them to EFI/Clover/Kexts/Other.

Make sure you do not use any of AirportBrcmFixup, BT4LEContinuityFixup, BrcmBluetoothInjector, BrcmPatchRAM3 so you do not create a conflict.

After reboot Bluetooth will appear in System Preferences and you will be able to find and pair your Bluetooth devices.

Wi-Fi driver

To get Intel Wi-Fi working you need itlwm. Similar to IntelBluetoothFirmware it is a macOS kernel extension using firmware from Linux.

Make sure your specific Intel card is supported and download the latest release. The release includes two kexts; itlwm.kext and itlwmx.kext. The itlwmx.kext is for use with the Intel X cards, like Intel X200, the itlwm.kext is for all the older cards like mine.

Networks management

When loaded, itlwm.kext makes your Intel Wi-Fi card available as an Ethernet card, not as a Wi-Fi card. This means you will not get the classic macOS user interface for connecting to Wi-Fi networks.

You need to either configure your Wi-Fi networks either manually or use a custom Wi-Fi management app.

To configure the Wi-Fi networks manually open itlwm.kext and find Info.plist. If you open Info.plist you will see a section called IOKitPersonalities:itlwm:WiFiConfig with 4 Wi-Fi networks configured. Just change it to your networks configuration, providing your networks names and passwords and save the changes.

In the iOS application I currently work on the users can choose a video from the device’s gallery and that video gets uploaded to the backend.

This functionality has always worked fine but recently somebody tried to upload s slow motion video and the application was not able to handle it.

Turns out slow motion videos need a special case, they work a bit differently than normal videos.

When you pick a video from the device’s gallery you get a PHAsset of type .video. You can use PHImageManager to load it as AVAsset.

The application just tried to cast it to AVURLAsset and processed it as a video file stored at some url that can be converted to Data and uploaded to the backend.

A slow motion video is an AVAsset, just not an AVURLAsset but an AVComposition and it needs to be treated differently.

The best way to make it work for my upload to backend scenario was to export it to a standard video file

guard let exportSession = AVAssetExportSession(asset: asset, presetName: AVAssetExportPresetMediumQuality) else {
    Log.error?.message("Could not create AVAssetExportSession")
    return
}

let targetURL = dataPathProvider.uploadsDirectory.appendingPathComponent("\(UUID().uuidString).mp4")

exportSession.outputURL = targetURL
exportSession.outputFileType = AVFileType.mp4
exportSession.shouldOptimizeForNetworkUse = true

exportSession.exportAsynchronously {
    let exportedAsset = AVURLAsset(url: targetURL)
    self.processVideoAsset(asset: exportedAsset)
}

After the slow motion video gets exported it can be converted to an AVURLAsset and treated the same way as a normal video file.