Asked  7 Months ago    Answers:  5   Viewed   35 times

I'm currently using PHP curl extension to communicate with some HTTP APIs.

I use a bulk loader to perform a lot of operations at once. The bulk endpoint must be called with POST method so I use a code like :

<?php
$h = curl_init($url);
curl_setopt(CURLOPT_POST, true);
curl_setopt(CURLOPT_POSTFIELDS, $data);
curl_setopt(CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($h);
curl_close($h);

The bulk endpoint allows me to send a huge amount of data (more than 200Mo at once). For the moment I need to load the data inside a variable which require PHP to be able to use enough memory...

I need to set memory_limit to a high value just for the bulk load...

Is there a way to use a file stream to send data with the curl PHP extension ? I saw the CURLOPT_INFILE and CURLOPT_READFUNCTION but it seems to don't work with POST method...

I also saw that the curl command line tool is able to perform a --data "@/path/to/file/content" which seems to be what I need...

Any ideas ?

 Answers

19

Use CURLOPT_INFILE

$curl = curl_init();
curl_setopt( $curl, CURLOPT_PUT, 1 );
curl_setopt( $curl, CURLOPT_INFILESIZE, filesize($tmpFile) );
curl_setopt( $curl, CURLOPT_INFILE, ($in=fopen($tmpFile, 'r')) );
curl_setopt( $curl, CURLOPT_CUSTOMREQUEST, 'POST' );
curl_setopt( $curl, CURLOPT_HTTPHEADER, [ 'Content-Type: application/json' ] );
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
$result = curl_exec($curl);
curl_close($curl);
fclose($in);
Wednesday, March 31, 2021
 
Novalirium
answered 7 Months ago
50

PUT

$data = array('username'=>'dog','password'=>'tall');
$data_json = json_encode($data);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json','Content-Length: ' . strlen($data_json)));
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS,$data_json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response  = curl_exec($ch);
curl_close($ch);

POST

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$data_json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response  = curl_exec($ch);
curl_close($ch);

GET See @Dan H answer

DELETE

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE");
curl_setopt($ch, CURLOPT_POSTFIELDS,$data_json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response  = curl_exec($ch);
curl_close($ch);
Wednesday, March 31, 2021
 
xenon
answered 7 Months ago
96

Use CURLOPT_CUSTOMREQUEST instead of CURLOPT_POST like;

curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "POST");

And one more thhing, you need to add

curl_setopt($curl, CURLOPT_POSTREDIR, 3);

3 means follow redirect with the same type of request both for 301 and 302 redirects.

By doing this, second request will be as POST request as well. Note that, CURLOPT_POSTREDIR implemented in PHP 5.3.2 here

Wednesday, March 31, 2021
 
Octopus
answered 7 Months ago
32

We have the classical consulting answer here: it depends. As the implementation of the method is store specific, we depend on the underlying store API. In case of JPA there's no chance to provide streaming access as ….getResultList() returns a List. Hence we also expose the List to the client as especially JPA developers might be used to working with lists. So for JPA the only option is using the pagination API.

For a store like Neo4j we support the streaming access as the repositories return Iterable on CRUD methods as well as on the execution of finder methods.

Saturday, September 25, 2021
 
DCD
answered 3 Weeks ago
DCD
41

Here's what I found:

In this script: https://docs.google.com/static/spreadsheets2/client/js/1150385833-codemirror.js

I found this function:

function onCopyCut(e) {
  if (!belongsToInput(e) || signalDOMEvent(cm, e))
    return;
  if (cm.somethingSelected()) {
    setLastCopied({
      lineWise: false,
      text: cm.getSelections()
    });
    if (e.type == "cut")
      cm.replaceSelection("", null, "cut")
  } else if (!cm.options.lineWiseCopyCut)
    return;
  else {
    var ranges = copyableRanges(cm);
    setLastCopied({
      lineWise: true,
      text: ranges.text
    });
    if (e.type == "cut")
      cm.operation(function() {
        cm.setSelections(ranges.ranges, 0, sel_dontScroll);
        cm.replaceSelection("", null, "cut")
      })
  }
  if (e.clipboardData) {
    e.clipboardData.clearData();
    var content = lastCopied.text.join("n");
    e.clipboardData.setData("Text", content);
    if (e.clipboardData.getData("Text") == content) {
      e.preventDefault();
      return
    }
  }
  var kludge = hiddenTextarea(),
    te = kludge.firstChild;
  cm.display.lineSpace.insertBefore(kludge, cm.display.lineSpace.firstChild);
  te.value = lastCopied.text.join("n");
  var hadFocus = document.activeElement;
  selectInput(te);
  setTimeout(function() {
    cm.display.lineSpace.removeChild(kludge);
    hadFocus.focus();
    if (hadFocus == div)
      input.showPrimarySelection()
  }, 50)
}

NEW FIND

I found that Google sheets loads this script:

(function() {
    window._docs_chrome_extension_exists = !0;
    window._docs_chrome_extension_features_version = 1;
    window._docs_chrome_extension_permissions = "alarms clipboardRead clipboardWrite identity power storage unlimitedStorage".split(" ");
}
).call(this);

This is tied to their own extensions

NEW FIND 2

When I paste in a cell, it uses these two functions:

Script: https://docs.google.com/static/spreadsheets2/client/js/1526657789-waffle_js_prod_core.js

p.B_a = function(a) {
  var b = a.Ge().clipboardData;
  if (b && (b = b.getData("text/plain"),
      !be(Kf(b)))) {
    b = Lm(b);
    var c = this.C.getRange(),
      d = this.C.getRange();
    d.jq() && $fc(this.Fd(), d) == this.getValue().length && (c = this.Fd(),
      d = c.childNodes.length,
      c = TJ(c, 0 < d && XJ(c.lastChild) ? d - 1 : d));
    c.yP(b);
    VJ(b, !1);
    a.preventDefault()
  }
};
p.Z1b = function() {
  var a = this.C.getRange();
  a && 1 < fec(a).textContent.length && SAc(this)
}

NEW FIND 3

This function is used when I select all and copy:

Script: https://docs.google.com/static/spreadsheets2/client/js/1526657789-waffle_js_prod_core.js

p.bxa = function(a, b) {
  this.D = b && b.Ge().clipboardData || null;
  this.J = !1;
  try {
    this.rda();
    if (this.D && "paste" == b.type) {
      var c = this.D,
        d = this.L,
        e = {},
        f = [];
      if (void 0 !== c.items)
        for (var h = c.items, k = 0; k < h.length; k++) {
          var l = h[k],
            n = l.type;
          f.push(n);
          if (!e[n] && d(n)) {
            a: switch (l.kind) {
              case "string":
                var q = xk(c.getData(l.type));
                break a;
              case "file":
                var t = l.getAsFile();
                q = t ? Bnd(t) : null;
                break a;
              default:
                q = null
            }
            var u = q;
            u && (e[n] = u)
          }
        }
      else {
        var z = c.types || [];
        for (h = 0; h < z.length; h++) {
          var E = z[h];
          f.push(E);
          !e[E] && d(E) && (e[E] = xk(c.getData(E)))
        }
        k = c.files || [];
        for (c = 0; c < k.length; c++) {
          u = k[c];
          var L = u.type;
          f.push(L);
          !e[L] && d(L) && (e[L] = Bnd(u))
        }
      }
      this.C = e;
      a: {
        for (d = 0; d < f.length; d++)
          if ("text/html" == f[d]) {
            var Q = !0;
            break a
          }
        Q = !1
      }
      this.H = Q || !And(f)
    }
    this.F.bxa(a, b);
    this.J && b.preventDefault()
  } finally {
    this.D = null
  }
}

ANSWER TO YOUR COMMENT

Here is the difference between e.clipboardData.setData() and execCommand("copy"):

e.clipboardData.setData() is used to manipulate the data going to the clipboard.

execCommand("copy") programatically calls the CMD/CTRL + C.

If you call execCommand("copy"), it will just copy your current selection just as if you pressed CMD/CTRL + C. You can also use this function with e.clipboardData.setData():

//Button being a HTML button element
button.addEventListener("click",function(){
  execCommand("copy");
});

//This function is called by a click or CMD/CTRL + C
window.addEventListener("copy",function(e){
  e.preventDefault();  
  e.clipboardData.setData("text/plain", "Hey!");
}

NEW FIND 3 (POSSIBLE ANSWER)

Don't use setTimeout for simulating long text because it will freeze up the UI. Instead, just use a large chunk of text.

This script works without timing out.

window.addEventListener('copy', function(e) {
  e.preventDefault();

  console.log("Started!");
  //This will throw an error on StackOverflow, but works on my website.
  //Use this to disable it for testing on StackOverflow
  //if (!(navigator.clipboard)) {
  if (navigator.clipboard) {
    document.getElementById("status").innerHTML = 'Copying, do not leave page.';
    document.getElementById("main").style.backgroundColor = '#BB595C';
    tryCopyAsync(e).then(() =>
      document.getElementById("main").style.backgroundColor = '#59BBB7',
      document.getElementById("status").innerHTML = 'Idle... Try copying',
      console.log('Copied!')
    );
  } else {
    console.log('Not async...');
    tryCopy(e);
    console.log('Copied!');
  }
});

function tryCopy(e) {
  e.clipboardData.setData("text/html", getText());
}
function getText() {
  var html = '';
  var row = '<div></div>';
  for (i = 0; i < 1000000; i++) {
    html += row;
  }
  return html;
}
async function tryCopyAsync(e) {
  navigator.clipboard.writeText(await getTextAsync());
}
async function getTextAsync() {
  var html = '';
  var row = '<div></div>';
  await waitNextFrame();
  for (i = 0; i < 1000000; i++) {
    html += row;
  }
  await waitNextFrame();
  html = [new ClipboardItem({"text/html": new Blob([html], {type: 'text/html'})})];
  return html;
}

//Credit: https://stackoverflow.com/a/66165276/7872728
function waitNextFrame() {
  return new Promise(postTask);
}

function postTask(task) {
  const channel = postTask.channel || new MessageChannel();
  channel.port1.addEventListener("message", () => task(), {
    once: true
  });
  channel.port2.postMessage("");
  channel.port1.start();
}
#main{
  width:100%;
  height:100vh;
  background:gray;
  color:white;
  font-weight:bold;
}
#status{
  text-align:center;
  padding-top:24px;
  font-size:16pt;
}
body{
  padding:0;
  margin:0;
  overflow:hidden;
}
<div id='main'>
  <div id='status'>Idle... Try copying</div>
</div>

To test, make sure you click inside the snippet before copying.

Full DEMO

window.addEventListener("load", function() {
  window.addEventListener("click", function() {
    hideCopying();
  });
  fallbackCopy = 0;
  if (navigator.permissions && navigator.permissions.query && notUnsupportedBrowser()) {
    navigator.permissions.query({
      name: 'clipboard-write'
    }).then(function(result) {
      if (result.state === 'granted') {
        clipboardAccess = 1;
      } else if (result.state === 'prompt') {
        clipboardAccess = 2;
      } else {
        clipboardAccess = 0;
      }
    });
  } else {
    clipboardAccess = 0;
  }
  window.addEventListener('copy', function(e) {
    if (fallbackCopy === 0) {
      showCopying();
      console.log("Started!");
      if (clipboardAccess > 0) {
        e.preventDefault();
        showCopying();
        tryCopyAsync(e).then(() =>
          hideCopying(),
          console.log('Copied! (Async)')
        );
      } else if (e.clipboardData) {
        e.preventDefault();
        console.log('Not async...');
        try {
          showCopying();
          tryCopy(e);
          console.log('Copied! (Not async)');
          hideCopying();
        } catch (error) {
          console.log(error.message);
        }
      } else {
        console.log('Not async fallback...');
        try {
          tryCopyFallback();
          console.log('Copied! (Fallback)');
        } catch (error) {
          console.log(error.message);
        }
        hideCopying();
      }
    } else {
      fallbackCopy = 0;
    }
  });
});

function notUnsupportedBrowser() {
  if (typeof InstallTrigger !== 'undefined') {
    return false;
  } else {
    return true;
  }
}

function tryCopyFallback() {
  var copyEl = document.createElement
  var body = document.body;
  var input = document.createElement("textarea");
  var text = getText();
  input.setAttribute('readonly', '');
  input.style.position = 'absolute';
  input.style.top = '-10000px';
  input.style.left = '-10000px';
  input.innerHTML = text;
  body.appendChild(input);
  input.focus();
  input.select();
  fallbackCopy = 1;
  document.execCommand("copy");
}

function hideCopying() {
  el("main").style.backgroundColor = '#59BBB7';
  el("status").innerHTML = 'Idle... Try copying';
}

function showCopying() {
  el("status").innerHTML = 'Copying, do not leave page.';
  el("main").style.backgroundColor = '#BB595C';
}

function el(a) {
  return document.getElementById(a);
}

function tryCopy(e) {
  e.clipboardData.setData("text/html", getText());
  e.clipboardData.setData("text/plain", getText());
}

function getText() {
  var html = '';
  var row = '<div></div>';
  for (i = 0; i < 1000000; i++) {
    html += row;
  }
  return html;
}
async function tryCopyAsync(e) {
  navigator.clipboard.write(await getTextAsync());
}
async function getTextAsync() {
  var html = '';
  var row = '<div></div>';
  await waitNextFrame();
  for (i = 0; i < 1000000; i++) {
    html += row;
  }
  await waitNextFrame();
  html = [new ClipboardItem({"text/html": new Blob([html], {type: 'text/html'}),"text/plain": new Blob([html], {type: 'text/plain'})})];
  return html;
}
//Credit: https://stackoverflow.com/a/66165276/7872728
function waitNextFrame() {
  return new Promise(postTask);
}

function postTask(task) {
  const channel = postTask.channel || new MessageChannel();
  channel.port1.addEventListener("message", () => task(), {
    once: true
  });
  channel.port2.postMessage("");
  channel.port1.start();
}
#main {
  width: 500px;
  height: 200px;
  background: gray;
  background: rgba(0, 0, 0, 0.4);
  color: white;
  font-weight: bold;
  margin-left: calc(50% - 250px);
  margin-top: calc(50vh - 100px);
  border-radius: 12px;
  border: 3px solid #fff;
  border: 3px solid rgba(0, 0, 0, 0.4);
  box-shadow: 5px 5px 50px -15px #000;
  box-shadow: 20px 20px 50px 15px rgba(0, 0, 0, 0.3);
}

#status {
  text-align: center;
  line-height: 180px;
  vertical-align: middle;
  font-size: 16pt;
}

body {
  background: lightgrey;
  background: linear-gradient(325deg, rgba(81, 158, 155, 1) 0%, rgba(157, 76, 79, 1) 100%);
  font-family: arial;
  height: 100vh;
  padding: 0;
  margin: 0;
  overflow: hidden;
}

@media only screen and (max-width: 700px) {
  #main {
    width: 100%;
    height: 100vh;
    border: 0;
    border-radius: 0;
    margin: 0;
  }

  #status {
    line-height: calc(100vh - 20px);
  }
}
<!DOCTYPE html>
<html>
  <head>
    <title>Clipboard Test</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta charset='UTF-8'>
  </head>
  <body>
    <div id='main'>
      <div id='status'>Click the webpage to start.</div>
    </div>
  </body>
</html>

DEMO webpage: Here is my DEMO

HELPFUL LINKS

  • web.dev/async-clipboard/
  • alligator.io/js/async-clipboard-api/
  • developer.mozilla.org/...
  • googlechrome.github.io/samples/async-clipboard/
  • caniuse.com/?search=clipboard
  • sitepoint.com/clipboard-api/
  • codepen.io alternative clipboard functions
  • w3schools.com (the old way)
Sunday, October 10, 2021
 
Mahmoud Saleh
answered 7 Days ago
Only authorized users can answer the question. Please sign in first, or register a free account.
Not the answer you're looking for? Browse other questions tagged :