Wednesday, July 17, 2013

How to copy files between sites using JavaScript REST in Office365 / SharePoint 2013

I’m currently playing with a POC for an App, and wanted to try to do the App as a SharePoint hosted one, only using JavaScript and REST.

The starting point was to call _vti_bin/ExcelRest.asmx on the host web from my app web, but this end-point does neither support CORS nor JSONP, so it can’t be used directly. My next thought was; Ok, let’s copy the file from the host web over to my app web, then call ExcelRest locally. Easier said than done!

While the final solution seems easy enough, the research, trial and error have taken me about 3 days. I’m now sharing this with you so you can spend your valuable time increasing the international GDP instead.

Note: If you want to copy files between two libraries on the same level, then you can use the copyTo method. http://server/site/_api/web/folders/GetByUrl('/site/srclib')/Files/getbyurl('madcow.xlsx')/copyTo(strNewUrl = '/site/targetlib/madcow.xlsx,bOverWrite = true)

Problem

Copy a file from a document library in one site to a document library in a different site using JavaScript and REST.
The code samples have URL’s using the App web proxy, but it’s easily modifiable for non-app work as well.

Step 1 – Reading the file

var hostweburl = decodeURIComponent(getParameterByName('SPHostUrl'));
var appweburl = decodeURIComponent(getParameterByName('SPAppWebUrl'));

var fileContentUrl = "_api/SP.AppContextSite(@target)/web/GetFileByServerRelativeUrl('/site/library/madcow.xlsx')/$value?@target='" + hostweburl + "'";

var executor = new SP.RequestExecutor(appweburl);
var info = {
    url: fileContentUrl,
    method: "GET",
    binaryStringResponseBody: true,
    success: function (data) {
        //binary data available in data.body
        var result = data.body;
    },
    error: function (err) {
        alert(JSON.stringify(err));
    }
};
executor.executeAsync(info);

The important parameter here is setting binaryStringResponseBody to true. Without this parameter the response is being decoded as UTF-8 and the result in the success callback is garbled data, which leads to a corrupt file on save.

The  binaryStringResponseBody parameter is not documented anywhere, but I stumbled upon binaryStringRequestbody in an msdn article which was used when uploading a file, and I figured it was worth a shot. Opening SP.RequestExecutor.debug.js I indeed found this parameter.

Step 2 – Patching SP.RequestExecutor.debug.js

Adding binaryStringResponseBody will upon return of the call cause a script error as seen in the figure below.

image


The method in question is reading over the response byte-by-byte from an Uint8Array, building a correctly encoded string. The issue is that it tries to concatenate to a variable named ret, which is not defined. The defined variable is named $v_0, and here we have a real bug in the script. The bug is there both in Office365 and SharePoint 2013 on-premise.

Luckily for us patching JavaScript is super easy. You merely override the methods involved somewhere in your own code before it’s being called. In the below sample it’s being called once the SP.RequestExecutor.js library has been loaded. The method named BinaryDecode is the one with the error, but you have to override more methods as the originator called is internalProcessXMLHttpRequestOnreadystatechange, and it cascades to calling other internal functions which can be renamed at random as the method names are autogenerated. (This happened for me today and I had to change just overrinding the first function).

$.getScript(scriptbase + "SP.RequestExecutor.js", function(){
SP.RequestExecutorInternalSharedUtility.BinaryDecode = function SP_RequestExecutorInternalSharedUtility$BinaryDecode(data) {
   var ret = '';

   if (data) {
      var byteArray = new Uint8Array(data);

      for (var i = 0; i < data.byteLength; i++) {
         ret = ret + String.fromCharCode(byteArray[i]);
      }
   }
   ;
   return ret;
};

SP.RequestExecutorUtility.IsDefined = function SP_RequestExecutorUtility$$1(data) {
   var nullValue = null;

   return data === nullValue || typeof data === 'undefined' || !data.length;
};

SP.RequestExecutor.ParseHeaders = function SP_RequestExecutor$ParseHeaders(headers) {
   if (SP.RequestExecutorUtility.IsDefined(headers)) {
      return null;
   }
   var result = {};
   var reSplit = new RegExp('\r?\n');
   var headerArray = headers.split(reSplit);

   for (var i = 0; i < headerArray.length; i++) {
      var currentHeader = headerArray[i];

      if (!SP.RequestExecutorUtility.IsDefined(currentHeader)) {
         var splitPos = currentHeader.indexOf(':');

         if (splitPos > 0) {
            var key = currentHeader.substr(0, splitPos);
            var value = currentHeader.substr(splitPos + 1);

            key = SP.RequestExecutorNative.trim(key);
            value = SP.RequestExecutorNative.trim(value);
            result[key.toUpperCase()] = value;
         }
      }
   }
   return result;
};

SP.RequestExecutor.internalProcessXMLHttpRequestOnreadystatechange = function SP_RequestExecutor$internalProcessXMLHttpRequestOnreadystatechange(xhr, requestInfo, timeoutId) {
   if (xhr.readyState === 4) {
      if (timeoutId) {
         window.clearTimeout(timeoutId);
      }
      xhr.onreadystatechange = SP.RequestExecutorNative.emptyCallback;
      var responseInfo = new SP.ResponseInfo();

      responseInfo.state = requestInfo.state;
      responseInfo.responseAvailable = true;
      if (requestInfo.binaryStringResponseBody) {
         responseInfo.body = SP.RequestExecutorInternalSharedUtility.BinaryDecode(xhr.response);
      }
      else {
         responseInfo.body = xhr.responseText;
      }
      responseInfo.statusCode = xhr.status;
      responseInfo.statusText = xhr.statusText;
      responseInfo.contentType = xhr.getResponseHeader('content-type');
      responseInfo.allResponseHeaders = xhr.getAllResponseHeaders();
      responseInfo.headers = SP.RequestExecutor.ParseHeaders(responseInfo.allResponseHeaders);
      if (xhr.status >= 200 && xhr.status < 300 || xhr.status === 1223) {
         if (requestInfo.success) {
            requestInfo.success(responseInfo);
         }
      }
      else {
         var error = SP.RequestExecutorErrors.httpError;
         var statusText = xhr.statusText;

         if (requestInfo.error) {
            requestInfo.error(responseInfo, error, statusText);
         }
      }
   }
};
}); 

Step 3 – Uploading the file

The next step is to save the file in a library on my app web. The crucial part again is to make sure the data is being treated as binary, this time with binaryStringRequestBody set to true. Make a note of the digest variable as well. On a page inheriting the SP masterpage you can get this value with $("#__REQUESTDIGEST").val(). If not then you have to execute a separate call to _api/contextinfo. The code for that is at the bottom of this post.

var appweburl = decodeURIComponent(getParameterByName('SPAppWebUrl'));
var executor = new SP.RequestExecutor(appweburl);
var info = {
    url: "_api/web/GetFolderByServerRelativeUrl('/appWebtargetFolder')/Files/Add(url='madcow.xlsx', overwrite=true)",
    method: "POST",
    headers: {
        "Accept": "application/json; odata=verbose",
        "X-RequestDigest": digest
    },
    contentType: "application/json;odata=verbose",
    binaryStringRequestBody: true,
    body: data.body,
    success: function(data) {
         alert("Success! Your file was uploaded to SharePoint.");
    },
    error: function (err) {
        alert("Oooooops... it looks like something went wrong uploading your file.");
    }
};
executor.executeAsync(info);

Journey

I started out using jQuery.ajax for my REST calls, but I did not manage to get the encoding right no matter how many posts I read on this. I read through a lot on the following links which led me to the final solution:

Get the digest value

$.ajax({
    url: "_api/contextinfo",
    type: "POST",
    contentType: "application/x-www-url-encoded",
    dataType: "json",
    headers: {
        "Accept": "application/json; odata=verbose",
    },
    success: function (data) {
        if (data.d) {
            var digest = data.d.GetContextWebInformation.FormDigestValue;
        }
    },
    error: function (err) {
        alert(JSON.stringify(err));
    }
});

33 comments:

  1. Very nice. I just read that MSDN article and did not see that. It still pays off to read those MSDN articles. Still confused why this does not work with ajax using a binaryStringRequestBody header value. Need to look at it further using fiddler.

    ReplyDelete
    Replies
    1. Hi,
      The question is probably how that property is being translated into a http call. I don't think the header is passed on directly. The SP.RequestExecutor object builds a JSON call which is passed on to the content window in a .postMessage. Take a look at SP.RequestExecutorInternalSharedUtility.$15

      Delete
  2. Seems the SP.RequestExecutorInternalSharedUtility.$14 function just got renamed on SPO as it's autogenerated. I'll look into a more permanent solution.

    ReplyDelete
    Replies
    1. Post is updated with overriding more methods to make it work even though method names change.

      Delete
  3. Nice article. At the top, you mention: "Note: If you want to copy files between two libraries on the same level, then you can use the copyTo method."

    Question: Is the copyTo method called using a POST, or a GET? Do you have a working example?

    Thanks.

    ReplyDelete
    Replies
    1. Hi,
      Here's the MSDN reference: http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spfile.copyto.aspx

      It's a method on SPFile, so you have to first get the file, then you can copy/move it. Don't have the code here right now.

      -m

      Delete
  4. Very nice post! However I have one question: Is it possible to upload a file/document to a remote site, using sp.webproxy.invoke? I mean can I upload a file from SharePoint to a specific asp.net web api website http://mysite.com/upload endpoint?

    Thanks in advance!

    ReplyDelete
    Replies
    1. Hi,
      If you first read in all the bytes to the browser using REST, then you certainly should be able to post this to some other web end-point. But you probably won't use SP.RequestExecutor. You can do a regular form post or use jquery to post your data.

      As long as your server code know how to handle the data being posted and which field it's in you're all good. And it's up to you to specify the format of the data you are posting.

      Delete
  5. I noticed in your executeAsync to send the file to SharePoint, you used "arrayBuffer" as your body. However, RequestExecutor.js enforces that the RequestInfo.body be typeof == "string", which would prevent you from sending an ArrayBuffer. Were you able to figure out how to get around this?

    Thanks.

    ReplyDelete
    Replies
    1. Hi,
      I haven't done anything special and I'm use the code above. My guess is that setting binaryStringRequestBody=true allows for binary data.

      Thanks,
      Mikael

      Delete
  6. very usefull ! thanks a lot for the trick !
    I just change the part wrapping the response body result into responInfo.body object cause I need the arrayBuffer result, not binary decoded :(into internalProcessXMLHttpRequestOnreadystatechange)

    if (requestInfo.binaryStringResponseBody) {
    responseInfo.body = xhr.response;
    }

    Thus, no more need to override BinaryDecode method...

    ReplyDelete
    Replies
    1. @paslatek can you please explain little bit more. i am not understanding what you have done. i am facing same problem which you are facing.

      Delete
  7. Thanks so much for posting! When I execute the script in my environment, I get an error: 'arrayBuffer' is underfined

    How did you build this variable?

    ReplyDelete
    Replies
    1. Never mind - I used data.body instead of arrayBuffer and that worked.

      Another alternative to posting the binary is to use jquery ajax, I used paslateks fix above to get the arrayBuffer and then used the ajax in Scott Hillier's blog (http://www.shillier.com/archive/2013/03/26/uploading-files-in-sharepoint-2013-using-csom-and-rest.aspx) to upload the file. This saves on performance because the script doesn't have to convert the array buffer into base64 binary string to upload.

      Kevin

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Edited my sample with this correction.

      Delete
  8. Have you tried this solution for larger file sizes (1-2gb)? I am getting an error from line 60 of your bug fix (SP.RequestExecutorInternalSharedUtility.BinaryDecode(xhr.response)) The value of xhr.response is "There is not enough storage to complete this operation."

    ReplyDelete
    Replies
    1. Hi,
      Since it's all going via the browser, I guess it's a browser issue. But bringing that much data via the browser seems like a bad idea to me.

      Delete
  9. This comment has been removed by the author.

    ReplyDelete
  10. I'm getting an error (There is no app context to execute this request). Is there a way around that?

    ReplyDelete
  11. Hi excellent post and just what I am looking for . In my case I converting your code into a series of Promises driven from a custom menu action app. which copies files between libraries in different HNSCs. Do you know know if the bug is still and issue.- will run the code tomorrow.

    ReplyDelete
  12. hi I have added embedded the code from this excellent post into 3 javascript promises The last promise is the one giving me the error. One thing I notice is that your file upload does reference hostweb as mine call does - but then my call gives me a WValue does not fall within expected range" "http://app-c12d92f281a882.appsdev.domain.local/appsdev/CopyTemplateApp/_api/SP.AppContextSite(@target)/web/GetFolderByServerRelativeUrl('/Shared Documents')/Files/Add(url='/stuff123.docx',overwrite=true)?@target='http://spdev/appsdev'" String

    ReplyDelete
  13. please send complete code I can't understand call methods names

    ReplyDelete
  14. I'm only trying to copy an attachment from a List to the Document Library of the same site. When I run the function, I get a FORBIDDEN error. I posted the code on SharePoint Stack Exchange.
    http://sharepoint.stackexchange.com/questions/142623/forbidden-error-when-trying-to-copyto-document-from-list-to-document-library

    Can you take a look and see where I am going wrong with this?
    Thanks.

    ReplyDelete
  15. Hi,

    I am working on SharePoint Online project and creating a SharePoint Designer workflow 2013 and want to use REST API. When I am creating workflow, getting "Access Denied" when moving to different sub-site in the same site collection.

    ReplyDelete
    Replies
    1. You need to make sure you are using the right access tokens etc in the call like explained at https://blog.appliedis.com/2014/10/09/sharepoint-designer-2013-workflow-working-with-web-services/

      Haven't done much REST myself with workflows as it's a bit cumbersome :)

      Delete
    2. Hi Vipul,

      I am also having issues with SP 2013 on prem designer workflow trying to use Rest API. Did you resolve this access denied error for cross site documents move ?

      Delete
  16. You made my day! Thanks for this post

    ReplyDelete
  17. Thanks for the post.

    I also just found another post explaining how to add a 'Binary data ajax transport for jQuery' so the binary data can be read correctly.

    Read the following post:

    http://www.henryalgus.com/reading-binary-files-using-jquery-ajax/

    Using this binary ajax transport, I can now copy a binary file (like Word document) from a document folder to an item list as attachment and open it in word online without getting errors about corrupted files

    ReplyDelete
  18. You can find another example here:

    http://www.n8d.at/blog/deploy-binary-files-from-sharepoint-hosted-app-to-host-web/

    ReplyDelete
  19. Hi Mikael,
    Your method works fine for uploading documents to different Site collection level but the problem is whatever document I upload the content inside it is missing. So I am not able to open the document.
    Please help me with this.
    Thanks in advance :)

    ReplyDelete