Building Earthquake API with Laravel
Building Earthquake API with Laravel 8.x

Laravel is the most powerful framework built with PHP. It has lots of ready-to-use features that you can use. You can visit the website to see all the details about it. Today I will use it to create an earthquake API service.

In this tutorial, I will create a data scraper to scrape earthquake data then create an API that serves earthquake data from Turkey.

I will follow these steps while I’m building this API service:

  1. Preparation
    1. Installing and Configuring Laravel
    2. Inspecting the Data Source
    3. Create Migrations to Create Database and Tables
    4. Create Models to Save, Serve and Manipulate Data
  2. Scraping Data for Earthquake API
  3. Creating Earthquake API
    1. Create a Controller to Serve Data
    2. Create Route for The API Interface

Preparation

Installing and Configuring Laravel

We need Composer to install Laravel. You can visit this URL to learn how to install it. Then we need to run this command in command line.

composer create-project laravel/laravel earthquake-api

This code will create a directory with name earthquake-api then put Laravel files into it.

I will create a database in MySQL named earthquake. If you do not have MySQL service on your computer or server you need to install it. You can check this URL on how to do it.

In .env we need to change MySQL parameters with ours. Like database name, MySQL username, and password. Here is how the .env file’s database section looks in the default

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=laravel
DB_USERNAME=root
DB_PASSWORD=

Inspecting the Data Source

Before creating the migration file we need to inspect what data we can scrape from the source website. This will define our columns to keep data. I checked that website then we need these columns like this:

Earthquakes data from Turkey
-- ----------------------------
-- Table structure for earthquakes
-- ----------------------------
DROP TABLE IF EXISTS `earthquakes`;
CREATE TABLE `earthquakes` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `latitude` double(8,2) NOT NULL,
  `longitude` double(8,2) NOT NULL,
  `depth` double(8,2) NOT NULL,
  `md` double(8,2) DEFAULT NULL,
  `ml` double(8,2) NOT NULL,
  `mw` double(8,2) DEFAULT NULL,
  `region` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `happened_at` timestamp NOT NULL,
  `hash` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
  `created_at` timestamp NULL DEFAULT NULL,
  `updated_at` timestamp NULL DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `hash` (`hash`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Create Migrations to Create Database and Tables

I will create just one migration file to keep the earthquakes. In the earthquake-api directory, we need to run this command to create a migration file that creates a database table which we need.

php artisan make:migration create_earthquakes_table
----------------------------------------------------
//Result : Created Migration: 2021_05_14_154256_create_earthquakes_table

We can find this file under earthquake-api/database/migrations/2021_05_14_154256_create_earthquakes_table.php

Here is the content of 2021_05_14_154256_create_earthquakes_table.php file

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class CreateEarthquakesTable extends Migration
{
    /**
     * Run the migrations.
     *
     * @return void
     */
    public function up()
    {
        Schema::create('earthquakes', function (Blueprint $table) {
            $table->id();
            $table->float('latitude');
            $table->float('longitude');
            $table->float('depth');
            $table->float('md')->nullable();//Generally empty
            $table->float('ml');
            $table->float('mw')->nullable();//Generally empty
            $table->string('region');
            $table->timestamp('happened_at');
            $table->string('hash')->index('hash');
            $table->timestamps();
        });
    }

    /**
     * Reverse the migrations.
     *
     * @return void
     */
    public function down()
    {
        Schema::dropIfExists('earthquakes');
    }
}

After editing this migration file we need to run a command that I wrote below. This command will run our migration files to create/update/delete etc. database tables.

php artisan migrate
----------------------------------------------------
Result: Migrated ........_table

This means all migrations ran successfully. So, the database tables were created successfully.

Create Models to Save, Serve and Manipulate Data

We need a model that named like “Earthquake”. You can run this command to create it.

php artisan make:model Models\Earthquake
----------------------------------------------------
Result: Model created successfully.

You can see this model file under earthquake-api/app/models/Earthquake.php

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Earthquake extends Model
{
    use HasFactory;

    //Link model with table
    protected $table = 'earthquakes';

    //Define fillable fields for model
    protected $fillable = [
        'latitude',
        'longitude',
        'depth',
        'md',
        'ml',
        'mw',
        'region',
        'happened_at',
        'hash',
    ];

}

Scraping Data for Earthquake API

In this step, we’ll inspect more deeply how to extract data from a remote source. We will use “command” classes to scrape data in the background. If you do not know about artisan commands in Laravel you should check this URL.

I will create a command with named ScrapeEarthquakes. I need to run this command to do it.

php artisan make:command ScrapeEarthquakes
----------------------------------------------------
Job created successfully.

We can find this file under earthquake-api/app/console/Commands/ScrapeEarthquakes.php. I will code my scraper code in this job file. Let’s code it.

Here is the content of ScrapeEarthquakes.php file

<?php

namespace App\Console\Commands;

use App\Models\Earthquake;
use Carbon\Carbon;
use Carbon\CarbonTimeZone;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Http;
use PHPHtmlParser\Dom;
use PHPHtmlParser\Options;

class ScrapeEarthquakes extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'scrape:earthquakes';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'This command tries to scrape earthquakes from http://www.koeri.boun.edu.tr/scripts/lasteq.asp';

    /**
     * Unwanted rows that will exclude from data.
     *
     * @var string[]
     */
    private $excluded = [
        'RECENT EARTHQUAKES IN TURKEY',
        'KOERI REGIONAL EARTHQUAKE-TSUNAMI MONITORING CENTER',
        '(QUICK EPICENTER DETERMINATIONS)',
        '                                                        Magnitude',
        'Date       Time      Latit(N)  Long(E)   Depth(km)     MD   ML   Mw    Region',
        '---------- --------  --------  -------   ----------    ------------    -----------',
    ];

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {

        //Format unwanted rows before use
        foreach ($this->excluded as $index => $row) {

            //Remove spaces before compare their equality
            $this->excluded[$index] = $this->clearSpaces($row);

        }

        parent::__construct();

    }

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {

        //Make a request to remote target html source which has data.
        $response = Http::get('http://www.koeri.boun.edu.tr/scripts/lasteq.asp');

        //Check is client successful
        if ($response->successful()) {

            //Create an instance from dom parser.
            $parser = new Dom();

            //Change options with that we need.
            $parser->setOptions(
                (new Options())
                    ->setWhitespaceTextNode(false)
                    ->setCleanupInput(false)
                    ->setPreserveLineBreaks(false)
                    ->setRemoveDoubleSpace(false)
            );

            //Load html source to dom parser
            $parser->loadStr($response->body());

            //Get raw data from html source
            $rawData = $parser->find('pre')[0]->innerHtml ?? false;

            //If is there raw data from source.
            if ($rawData) {

                //Explode raw data to rows
                $rows = explode(PHP_EOL, $rawData);

                //Process all rows with progressbar.
                $this->withProgressBar($rows, function ($row) {
                    $this->processEarthquake($row);
                });

            }

        }

    }

    /**
     * @param $row
     */
    protected function processEarthquake($row)
    {

        //Check row if it is unwanted row or empty

        if (!empty($this->clearSpaces($row)) && !in_array($this->clearSpaces($row), $this->excluded)) {

            //Replace the row's double spaces with # to extract data easier
            $row = $this->convertDoubleSpaces($row);

            //Slice the row with #
            $rowParts = explode('#', $row);

            //A temporary variable to keep row data as columns
            $earthquakeData = [];

            //Process all sliced row parts
            foreach ($rowParts as $part) {

                //Remove spaces from left and right like " hello " to "hello"
                $part = trim($part);

                //Check if part is not empty or not equals # then add it as data
                if ($part !== '#' && !empty($part)) {
                    $earthquakeData[] = $part;
                }

            }

            //Check is not empty array, this means row has data
            if (count($earthquakeData)){

                //Create a hash value to better check
                $earthquakeHash = hash('crc32', implode(',', $earthquakeData));

                //Create a carbon instance from locale formatted date time
                $happenedAt = Carbon::createFromFormat('Y.m.d H:i:s', $earthquakeData[0], new CarbonTimeZone('Europe/Istanbul'));

                //Check if it is not already saved
                if (!Earthquake::where('hash', $earthquakeHash)->count()){

                    Earthquake::create([
                        'latitude' => $earthquakeData[1],
                        'longitude' => $earthquakeData[2],
                        'depth' => $earthquakeData[3],
                        'md' => ($earthquakeData[4] !== '-.-' ? $earthquakeData[4] : null),
                        'ml' => $earthquakeData[5],
                        'mw' => ($earthquakeData[6] !== '-.-' ? $earthquakeData[6] : null),
                        'region' => $earthquakeData[7],
                        'happened_at' => $happenedAt,
                        'hash' => $earthquakeHash,
                    ]);

                }

            }

        }

    }

    /**
     * @param $text string The text which we want to clear spaces
     * @return string
     */

    protected function clearSpaces($text)
    {
        return str_replace(' ', '', trim($text));
    }

    /**
     * @param $text string The text which we want to clear
     * @return string
     */

    protected function convertDoubleSpaces($text)
    {
        return str_replace('  ', '##', trim($text));
    }

}

With this command file, we can run this process with a command line or cronjob. We can run this command with the command below

php artisan scrape:earthquakes

As you can see the results in the image below. These rows came from our scraper. We need to create a cronjob to repeat this command. In cPanel under cronjobs you can use this command:

php /home/[username]/public_html/artisan scrape:earthquakes
Scraping results from the database

Creating Earthquake API

Create a Controller to Serve Data

We need a controller file that serves our earthquake data to users. You need to run this command to create a controller file.

php artisan make:controller EarthquakeController
----------------------------------------------------
Controller created successfully.

You can find this file under earthquake-api/app/Http/Controllers/EarthquakeController.php. Here is my code that I wrote for the controller.

<?php

namespace App\Http\Controllers;

use App\Models\Earthquake;
use Illuminate\Http\Request;

class EarthquakeController extends Controller
{

    public function index(Request $request)
    {
        
        //Accessing earthquake model instance to run mysql queries.
        $response = Earthquake::orderBy('happened_at', 'desc')
            ->when($request->get('search'), function ($query, $search) {
                //When request has search parameter apply this filter.
                $query->where('region', 'like', '%' . $search . '%');
            })->simplePaginate();//Automatically creates pagination
        
        //return this model object as response. Laravel will convert this object to json response automatically.
        return response($response);
      
    }
    
}

Create Route for The API Interface

This is the last step for this tutorial. We need to create a routing rule to serve data from URL. Here is my API route rule file. You can find this file under earthquake-api/routes/api.php

<?php

use App\Http\Controllers\EarthquakeController;
use Illuminate\Support\Facades\Route;

/*
|--------------------------------------------------------------------------
| API Routes
|--------------------------------------------------------------------------
|
| Here is where you can register API routes for your application. These
| routes are loaded by the RouteServiceProvider within a group which
| is assigned the "api" middleware group. Enjoy building your API!
|
*/

Route::get('/', [EarthquakeController::class, 'index']);

With this step by step tutorial is finished. You can test your api service with a test server that you can setup by running this code to setup a web server.

php artisan serve
----------------------------------------------------
PHP 8.0.3 Development Server (http://127.0.0.1:8003) started

I hope you enjoy this tutorial.

If you got any questions you can ask me in comments section below.

See you in text post!