Organizational Research By

Surprising Reserch Topic

module request how to properly retrieve accented characters

module request how to properly retrieve accented characters  using -'node.js,request,web-scraping'

I'm using: Module: Request -- Simplified HTTP request method to scrape a webpage with accented characters á é ó ú ê ã etc.

I've already tried encoding: utf-8 with no success. I'm still getting this ��� characters in the result.

    uri: url,
    encoding: 'utf-8'
    // ...

Is there any configuration to fix it?

I don't know if it is an issue, but I filled one for this module. No answers yet. :/

asked Oct 7, 2015 by balvant maurya
0 votes

Related Hot Questions

2 Answers

0 votes

Specify the encoding as utf8 not utf-8. Here are a list of possible encodings for a buffer from the Node.js documentation.

  • ascii - for 7 bit ASCII data only. This encoding method is very fast, and will strip the high bit if set.
  • utf8 - Unicode characters. Many web pages and other document formats use UTF-8.
  • base64 - Base64 string encoding.
  • 'binary - A way of encoding raw binary data into strings by using only the first 8 bits of each character. This encoding method is depreciated and should be avoided in favor of Buffer objects where possible. This encoding will be removed in future versions of Node.
answered Oct 7, 2015 by ukohale
0 votes

Since binary is deprecated it seems like a better idea to use iconv and correctly handle the decoding:

var request = require("request"), iconv  = require('iconv-lite');
var requestOptions  = { encoding: null, method: "GET", uri: ""};

request(requestOptions, function(error, response, body) {
    var utf8String = iconv.decode(new Buffer(body), "ISO-8859-1");

The important part is to set the encoding on the HTTP request to be null encoding: null.

answered Oct 7, 2015 by sumit_jaiswalmca