Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote206 views1 answer
Required depend filed odoo
Python
0 vote219 views1 answer
Module 'tensorflow' has no attribute 'InteractiveSession'
Python
0 vote254 views1 answer
Migrations.rename_field django
Python Django
0 vote217 views1 answer
Create a fibonacci function using a generator python
Python
0 vote207 views1 answer
Add new dimension
Python
0 vote217 views2 answers
Python error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Python
11 votes202 views2 answers
Python functions with input
Python
10 votes213 views5 answers
Opencv Webcam
Python
1 vote208 views1 answer
Where my python modules in linux
Python
0 vote182 views1 answer
Keras.utils.plot_model keeps telling me to install pydot and graphviz
Python